Note: This is test shard 1 of 3.
[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250114 02:53:54.218117 429 test_util.cc:274] Using random seed: -1410742282
2025-01-14T02:53:54Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-14T02:53:54Z Disabled control of system clock
I20250114 02:53:54.294276 429 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrCUXNq/build/tsan/bin/kudu
/tmp/dist-test-taskrCUXNq/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.107.126:45367
--webserver_interface=127.0.107.126
--webserver_port=0
--builtin_ntp_servers=127.0.107.84:40455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.107.126:45367,127.0.107.125:46837,127.0.107.124:39105
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250114 02:53:54.600459 438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 02:53:54.601001 438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 02:53:54.601392 438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 02:53:54.632385 438 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250114 02:53:54.632668 438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 02:53:54.632879 438 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250114 02:53:54.633072 438 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250114 02:53:54.668376 438 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.107.84:40455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.107.126:45367,127.0.107.125:46837,127.0.107.124:39105
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.107.126:45367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.107.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type FASTDEBUG
built by None at 14 Jan 2025 02:43:23 UTC on a633016ff5c9
build id 4168
TSAN enabled
I20250114 02:53:54.669734 438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 02:53:54.671408 438 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 02:53:54.688957 445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 02:53:54.689368 444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 02:53:54.691219 438 server_base.cc:1034] running on GCE node
W20250114 02:53:54.690295 447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 02:53:55.873589 438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 02:53:55.876940 438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 02:53:55.878298 438 hybrid_clock.cc:648] HybridClock initialized: now 1736823235878245 us; error 67 us; skew 500 ppm
I20250114 02:53:55.879077 438 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250114 02:53:55.892081 438 webserver.cc:458] Webserver started at http://127.0.107.126:40319/ using document root <none> and password file <none>
I20250114 02:53:55.893080 438 fs_manager.cc:362] Metadata directory not provided
I20250114 02:53:55.893306 438 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 02:53:55.893734 438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 02:53:55.898522 438 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-0/data/instance:
uuid: "b303b7df086340a0b5ad6373940ae750"
format_stamp: "Formatted at 2025-01-14 02:53:55 on dist-test-slave-36vg"
I20250114 02:53:55.899581 438 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-0/wal/instance:
uuid: "b303b7df086340a0b5ad6373940ae750"
format_stamp: "Formatted at 2025-01-14 02:53:55 on dist-test-slave-36vg"
I20250114 02:53:55.906468 438 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20250114 02:53:55.912120 454 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 02:53:55.913107 438 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20250114 02:53:55.913399 438 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-0/data,/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-0/wal
uuid: "b303b7df086340a0b5ad6373940ae750"
format_stamp: "Formatted at 2025-01-14 02:53:55 on dist-test-slave-36vg"
I20250114 02:53:55.913739 438 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 02:53:55.963604 438 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 02:53:55.965005 438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 02:53:55.965440 438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 02:53:56.043187 438 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.126:45367
I20250114 02:53:56.043254 505 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.126:45367 every 8 connection(s)
I20250114 02:53:56.045907 438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-0/data/info.pb
I20250114 02:53:56.051466 429 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrCUXNq/build/tsan/bin/kudu as pid 438
I20250114 02:53:56.051997 429 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-0/wal/instance
I20250114 02:53:56.052745 506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:56.057420 429 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrCUXNq/build/tsan/bin/kudu
/tmp/dist-test-taskrCUXNq/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.107.125:46837
--webserver_interface=127.0.107.125
--webserver_port=0
--builtin_ntp_servers=127.0.107.84:40455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.107.126:45367,127.0.107.125:46837,127.0.107.124:39105
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250114 02:53:56.067564 506 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.107.126" port: 45367 } has no permanent_uuid. Determining permanent_uuid...
I20250114 02:53:56.092206 506 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } has no permanent_uuid. Determining permanent_uuid...
W20250114 02:53:56.095007 455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.107.125:46837: connect: Connection refused (error 111)
W20250114 02:53:56.099126 506 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.107.125:46837: Network error: Client connection negotiation failed: client connection to 127.0.107.125:46837: connect: Connection refused (error 111)
I20250114 02:53:56.148552 506 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } attempt: 1
W20250114 02:53:56.152758 506 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.107.125:46837: Network error: Client connection negotiation failed: client connection to 127.0.107.125:46837: connect: Connection refused (error 111)
I20250114 02:53:56.221092 506 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } attempt: 2
W20250114 02:53:56.225075 506 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.107.125:46837: Network error: Client connection negotiation failed: client connection to 127.0.107.125:46837: connect: Connection refused (error 111)
I20250114 02:53:56.316423 506 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } attempt: 3
W20250114 02:53:56.320331 506 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.107.125:46837: Network error: Client connection negotiation failed: client connection to 127.0.107.125:46837: connect: Connection refused (error 111)
W20250114 02:53:56.364461 508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 02:53:56.365047 508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 02:53:56.365492 508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 02:53:56.396314 508 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250114 02:53:56.396637 508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 02:53:56.396919 508 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250114 02:53:56.397158 508 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250114 02:53:56.431382 508 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.107.84:40455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.107.126:45367,127.0.107.125:46837,127.0.107.124:39105
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.107.125:46837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.107.125
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type FASTDEBUG
built by None at 14 Jan 2025 02:43:23 UTC on a633016ff5c9
build id 4168
TSAN enabled
I20250114 02:53:56.432783 508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 02:53:56.434294 508 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 02:53:56.451869 517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 02:53:56.463739 506 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } attempt: 4
W20250114 02:53:56.468674 506 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.107.125:46837: Network error: Client connection negotiation failed: client connection to 127.0.107.125:46837: connect: Connection refused (error 111)
I20250114 02:53:56.768182 506 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } attempt: 5
W20250114 02:53:56.772672 506 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.107.125:46837: Network error: Client connection negotiation failed: client connection to 127.0.107.125:46837: connect: Connection refused (error 111)
I20250114 02:53:57.320370 506 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } attempt: 6
W20250114 02:53:57.330953 506 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.107.125:46837: Network error: Client connection negotiation failed: client connection to 127.0.107.125:46837: connect: Connection refused (error 111)
W20250114 02:53:56.452729 518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 02:53:56.454325 508 server_base.cc:1034] running on GCE node
W20250114 02:53:56.453629 520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 02:53:57.668411 508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 02:53:57.670899 508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 02:53:57.672291 508 hybrid_clock.cc:648] HybridClock initialized: now 1736823237672239 us; error 38 us; skew 500 ppm
I20250114 02:53:57.673130 508 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250114 02:53:57.680095 508 webserver.cc:458] Webserver started at http://127.0.107.125:37011/ using document root <none> and password file <none>
I20250114 02:53:57.681059 508 fs_manager.cc:362] Metadata directory not provided
I20250114 02:53:57.681269 508 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 02:53:57.681713 508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 02:53:57.686574 508 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-1/data/instance:
uuid: "687d4ff6d3cb42cf90a965070b0d8a70"
format_stamp: "Formatted at 2025-01-14 02:53:57 on dist-test-slave-36vg"
I20250114 02:53:57.687688 508 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-1/wal/instance:
uuid: "687d4ff6d3cb42cf90a965070b0d8a70"
format_stamp: "Formatted at 2025-01-14 02:53:57 on dist-test-slave-36vg"
I20250114 02:53:57.695458 508 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20250114 02:53:57.701323 529 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 02:53:57.702304 508 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20250114 02:53:57.702610 508 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-1/data,/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-1/wal
uuid: "687d4ff6d3cb42cf90a965070b0d8a70"
format_stamp: "Formatted at 2025-01-14 02:53:57 on dist-test-slave-36vg"
I20250114 02:53:57.702960 508 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 02:53:57.749508 508 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 02:53:57.750900 508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 02:53:57.751329 508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 02:53:57.826949 508 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.125:46837
I20250114 02:53:57.827041 580 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.125:46837 every 8 connection(s)
I20250114 02:53:57.829579 508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-1/data/info.pb
I20250114 02:53:57.832139 429 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrCUXNq/build/tsan/bin/kudu as pid 508
I20250114 02:53:57.832762 429 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-1/wal/instance
I20250114 02:53:57.835150 429 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrCUXNq/build/tsan/bin/kudu
/tmp/dist-test-taskrCUXNq/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.107.124:39105
--webserver_interface=127.0.107.124
--webserver_port=0
--builtin_ntp_servers=127.0.107.84:40455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.107.126:45367,127.0.107.125:46837,127.0.107.124:39105
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250114 02:53:57.835662 581 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:57.847375 581 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.107.126" port: 45367 } has no permanent_uuid. Determining permanent_uuid...
I20250114 02:53:57.869354 581 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } has no permanent_uuid. Determining permanent_uuid...
I20250114 02:53:57.884779 581 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } has no permanent_uuid. Determining permanent_uuid...
W20250114 02:53:57.886889 530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.107.124:39105: connect: Connection refused (error 111)
W20250114 02:53:57.891680 581 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.107.124:39105: Network error: Client connection negotiation failed: client connection to 127.0.107.124:39105: connect: Connection refused (error 111)
I20250114 02:53:57.941161 581 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } attempt: 1
W20250114 02:53:57.945331 581 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.107.124:39105: Network error: Client connection negotiation failed: client connection to 127.0.107.124:39105: connect: Connection refused (error 111)
I20250114 02:53:58.013689 581 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } attempt: 2
W20250114 02:53:58.017819 581 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.107.124:39105: Network error: Client connection negotiation failed: client connection to 127.0.107.124:39105: connect: Connection refused (error 111)
I20250114 02:53:58.109225 581 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } attempt: 3
W20250114 02:53:58.113209 581 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.107.124:39105: Network error: Client connection negotiation failed: client connection to 127.0.107.124:39105: connect: Connection refused (error 111)
W20250114 02:53:58.153512 583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 02:53:58.154083 583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 02:53:58.154479 583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 02:53:58.185175 583 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250114 02:53:58.185456 583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 02:53:58.185719 583 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250114 02:53:58.185912 583 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250114 02:53:58.220860 583 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.107.84:40455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.107.126:45367,127.0.107.125:46837,127.0.107.124:39105
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.107.124:39105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.107.124
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type FASTDEBUG
built by None at 14 Jan 2025 02:43:23 UTC on a633016ff5c9
build id 4168
TSAN enabled
I20250114 02:53:58.222187 583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 02:53:58.223747 583 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 02:53:58.240108 594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 02:53:58.256711 581 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } attempt: 4
W20250114 02:53:58.260957 581 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.107.124:39105: Network error: Client connection negotiation failed: client connection to 127.0.107.124:39105: connect: Connection refused (error 111)
I20250114 02:53:58.391379 506 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } attempt: 7
I20250114 02:53:58.403589 506 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } has no permanent_uuid. Determining permanent_uuid...
W20250114 02:53:58.410001 506 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.107.124:39105: Network error: Client connection negotiation failed: client connection to 127.0.107.124:39105: connect: Connection refused (error 111)
I20250114 02:53:58.468392 506 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } attempt: 1
W20250114 02:53:58.473235 506 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.107.124:39105: Network error: Client connection negotiation failed: client connection to 127.0.107.124:39105: connect: Connection refused (error 111)
I20250114 02:53:58.554659 506 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } attempt: 2
W20250114 02:53:58.559015 506 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.107.124:39105: Network error: Client connection negotiation failed: client connection to 127.0.107.124:39105: connect: Connection refused (error 111)
I20250114 02:53:58.560539 581 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } attempt: 5
W20250114 02:53:58.564661 581 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.107.124:39105: Network error: Client connection negotiation failed: client connection to 127.0.107.124:39105: connect: Connection refused (error 111)
I20250114 02:53:58.644390 506 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } attempt: 3
W20250114 02:53:58.648715 506 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.107.124:39105: Network error: Client connection negotiation failed: client connection to 127.0.107.124:39105: connect: Connection refused (error 111)
I20250114 02:53:58.789173 506 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } attempt: 4
W20250114 02:53:58.793954 506 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.107.124:39105: Network error: Client connection negotiation failed: client connection to 127.0.107.124:39105: connect: Connection refused (error 111)
I20250114 02:53:59.077555 506 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } attempt: 5
W20250114 02:53:59.081597 506 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.107.124:39105: Network error: Client connection negotiation failed: client connection to 127.0.107.124:39105: connect: Connection refused (error 111)
I20250114 02:53:59.112277 581 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } attempt: 6
W20250114 02:53:59.119427 581 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.107.124:39105: Network error: Client connection negotiation failed: client connection to 127.0.107.124:39105: connect: Connection refused (error 111)
W20250114 02:53:58.240908 593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 02:53:58.243350 596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 02:53:58.243981 583 server_base.cc:1034] running on GCE node
I20250114 02:53:59.433537 583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 02:53:59.436028 583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 02:53:59.437450 583 hybrid_clock.cc:648] HybridClock initialized: now 1736823239437405 us; error 65 us; skew 500 ppm
I20250114 02:53:59.438251 583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250114 02:53:59.444653 583 webserver.cc:458] Webserver started at http://127.0.107.124:36399/ using document root <none> and password file <none>
I20250114 02:53:59.445677 583 fs_manager.cc:362] Metadata directory not provided
I20250114 02:53:59.445886 583 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 02:53:59.446282 583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 02:53:59.451058 583 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-2/data/instance:
uuid: "87b3db09ac87436e9a60d6de5fedaff5"
format_stamp: "Formatted at 2025-01-14 02:53:59 on dist-test-slave-36vg"
I20250114 02:53:59.452037 583 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-2/wal/instance:
uuid: "87b3db09ac87436e9a60d6de5fedaff5"
format_stamp: "Formatted at 2025-01-14 02:53:59 on dist-test-slave-36vg"
I20250114 02:53:59.459677 583 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20250114 02:53:59.465766 608 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 02:53:59.466773 583 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250114 02:53:59.467083 583 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-2/data,/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-2/wal
uuid: "87b3db09ac87436e9a60d6de5fedaff5"
format_stamp: "Formatted at 2025-01-14 02:53:59 on dist-test-slave-36vg"
I20250114 02:53:59.467419 583 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 02:53:59.524184 583 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 02:53:59.525653 583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 02:53:59.526079 583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 02:53:59.602869 583 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.124:39105
I20250114 02:53:59.602957 659 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.124:39105 every 8 connection(s)
I20250114 02:53:59.605557 583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-2/data/info.pb
I20250114 02:53:59.609167 429 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrCUXNq/build/tsan/bin/kudu as pid 583
I20250114 02:53:59.609651 429 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-2/wal/instance
I20250114 02:53:59.612483 660 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:53:59.630299 660 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.107.126" port: 45367 } has no permanent_uuid. Determining permanent_uuid...
I20250114 02:53:59.634136 506 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } attempt: 6
I20250114 02:53:59.669958 660 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } has no permanent_uuid. Determining permanent_uuid...
I20250114 02:53:59.674983 506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Bootstrap starting.
I20250114 02:53:59.681105 660 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } has no permanent_uuid. Determining permanent_uuid...
I20250114 02:53:59.682313 506 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:59.684396 506 log.cc:826] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Log is configured to *not* fsync() on all Append() calls
I20250114 02:53:59.691846 506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: No bootstrap required, opened a new log
I20250114 02:53:59.701571 660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5: Bootstrap starting.
I20250114 02:53:59.709463 660 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5: Neither blocks nor log segments found. Creating new log.
I20250114 02:53:59.712414 660 log.cc:826] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5: Log is configured to *not* fsync() on all Append() calls
I20250114 02:53:59.718963 660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5: No bootstrap required, opened a new log
I20250114 02:53:59.722872 506 raft_consensus.cc:357] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b303b7df086340a0b5ad6373940ae750" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 45367 } } peers { permanent_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } } peers { permanent_uuid: "87b3db09ac87436e9a60d6de5fedaff5" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } }
I20250114 02:53:59.723766 506 raft_consensus.cc:383] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:59.724071 506 raft_consensus.cc:738] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b303b7df086340a0b5ad6373940ae750, State: Initialized, Role: FOLLOWER
I20250114 02:53:59.724977 506 consensus_queue.cc:260] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b303b7df086340a0b5ad6373940ae750" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 45367 } } peers { permanent_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } } peers { permanent_uuid: "87b3db09ac87436e9a60d6de5fedaff5" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } }
I20250114 02:53:59.728686 672 sys_catalog.cc:455] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b303b7df086340a0b5ad6373940ae750" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 45367 } } peers { permanent_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } } peers { permanent_uuid: "87b3db09ac87436e9a60d6de5fedaff5" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } } }
I20250114 02:53:59.729326 672 sys_catalog.cc:458] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750 [sys.catalog]: This master's current role is: FOLLOWER
I20250114 02:53:59.730756 506 sys_catalog.cc:564] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750 [sys.catalog]: configured and running, proceeding with master startup.
I20250114 02:53:59.749836 660 raft_consensus.cc:357] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b303b7df086340a0b5ad6373940ae750" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 45367 } } peers { permanent_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } } peers { permanent_uuid: "87b3db09ac87436e9a60d6de5fedaff5" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } }
I20250114 02:53:59.750869 660 raft_consensus.cc:383] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:53:59.751205 660 raft_consensus.cc:738] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87b3db09ac87436e9a60d6de5fedaff5, State: Initialized, Role: FOLLOWER
I20250114 02:53:59.752755 660 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b303b7df086340a0b5ad6373940ae750" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 45367 } } peers { permanent_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } } peers { permanent_uuid: "87b3db09ac87436e9a60d6de5fedaff5" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } }
I20250114 02:53:59.757493 684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b303b7df086340a0b5ad6373940ae750" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 45367 } } peers { permanent_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } } peers { permanent_uuid: "87b3db09ac87436e9a60d6de5fedaff5" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } } }
I20250114 02:53:59.758656 684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [sys.catalog]: This master's current role is: FOLLOWER
I20250114 02:53:59.759150 660 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [sys.catalog]: configured and running, proceeding with master startup.
W20250114 02:53:59.763381 683 catalog_manager.cc:1559] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250114 02:53:59.763859 683 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250114 02:53:59.782271 614 tablet.cc:2367] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250114 02:53:59.765110 438 master_runner.cc:186] Error getting master registration for 127.0.107.125:46837: OK, instance_id { permanent_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" instance_seqno: 1736823237806267 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250114 02:53:59.802717 438 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250114 02:53:59.811923 696 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250114 02:53:59.812448 696 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250114 02:53:59.808357 583 master_runner.cc:186] Error getting master registration for 127.0.107.125:46837: OK, instance_id { permanent_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" instance_seqno: 1736823237806267 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250114 02:53:59.842161 583 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250114 02:53:59.975034 460 tablet.cc:2367] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250114 02:54:00.040005 672 raft_consensus.cc:491] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:00.040585 672 raft_consensus.cc:513] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b303b7df086340a0b5ad6373940ae750" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 45367 } } peers { permanent_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } } peers { permanent_uuid: "87b3db09ac87436e9a60d6de5fedaff5" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } }
I20250114 02:54:00.046602 635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b303b7df086340a0b5ad6373940ae750" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87b3db09ac87436e9a60d6de5fedaff5" is_pre_election: true
I20250114 02:54:00.047574 635 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b303b7df086340a0b5ad6373940ae750 in term 0.
I20250114 02:54:00.049119 455 leader_election.cc:304] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87b3db09ac87436e9a60d6de5fedaff5, b303b7df086340a0b5ad6373940ae750; no voters:
I20250114 02:54:00.051556 672 leader_election.cc:290] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 687d4ff6d3cb42cf90a965070b0d8a70 (127.0.107.125:46837), 87b3db09ac87436e9a60d6de5fedaff5 (127.0.107.124:39105)
I20250114 02:54:00.052583 672 raft_consensus.cc:2798] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:00.053205 672 raft_consensus.cc:491] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:00.053828 672 raft_consensus.cc:3054] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:00.078202 672 raft_consensus.cc:513] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b303b7df086340a0b5ad6373940ae750" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 45367 } } peers { permanent_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } } peers { permanent_uuid: "87b3db09ac87436e9a60d6de5fedaff5" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } }
I20250114 02:54:00.084022 635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b303b7df086340a0b5ad6373940ae750" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87b3db09ac87436e9a60d6de5fedaff5"
I20250114 02:54:00.084638 635 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:00.091450 635 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b303b7df086340a0b5ad6373940ae750 in term 1.
I20250114 02:54:00.092618 455 leader_election.cc:304] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87b3db09ac87436e9a60d6de5fedaff5, b303b7df086340a0b5ad6373940ae750; no voters:
I20250114 02:54:00.093822 698 raft_consensus.cc:2798] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:00.097150 672 leader_election.cc:290] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750 [CANDIDATE]: Term 1 election: Requested vote from peers 687d4ff6d3cb42cf90a965070b0d8a70 (127.0.107.125:46837), 87b3db09ac87436e9a60d6de5fedaff5 (127.0.107.124:39105)
I20250114 02:54:00.103322 698 raft_consensus.cc:695] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750 [term 1 LEADER]: Becoming Leader. State: Replica: b303b7df086340a0b5ad6373940ae750, State: Running, Role: LEADER
I20250114 02:54:00.105886 698 consensus_queue.cc:237] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b303b7df086340a0b5ad6373940ae750" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 45367 } } peers { permanent_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } } peers { permanent_uuid: "87b3db09ac87436e9a60d6de5fedaff5" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } }
I20250114 02:54:00.129925 672 sys_catalog.cc:455] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b303b7df086340a0b5ad6373940ae750. Latest consensus state: current_term: 1 leader_uuid: "b303b7df086340a0b5ad6373940ae750" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b303b7df086340a0b5ad6373940ae750" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 45367 } } peers { permanent_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } } peers { permanent_uuid: "87b3db09ac87436e9a60d6de5fedaff5" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } } }
I20250114 02:54:00.130818 672 sys_catalog.cc:458] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750 [sys.catalog]: This master's current role is: LEADER
I20250114 02:54:00.135512 701 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250114 02:54:00.148195 701 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250114 02:54:00.176647 635 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [term 1 FOLLOWER]: Refusing update from remote peer b303b7df086340a0b5ad6373940ae750: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 02:54:00.180040 581 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } attempt: 7
I20250114 02:54:00.179946 698 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87b3db09ac87436e9a60d6de5fedaff5" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:00.225975 684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b303b7df086340a0b5ad6373940ae750. Latest consensus state: current_term: 1 leader_uuid: "b303b7df086340a0b5ad6373940ae750" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b303b7df086340a0b5ad6373940ae750" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 45367 } } peers { permanent_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } } peers { permanent_uuid: "87b3db09ac87436e9a60d6de5fedaff5" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } } }
I20250114 02:54:00.228343 684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [sys.catalog]: This master's current role is: FOLLOWER
I20250114 02:54:00.238366 699 sys_catalog.cc:455] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b303b7df086340a0b5ad6373940ae750" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b303b7df086340a0b5ad6373940ae750" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 45367 } } peers { permanent_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } } peers { permanent_uuid: "87b3db09ac87436e9a60d6de5fedaff5" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } } }
I20250114 02:54:00.239195 699 sys_catalog.cc:458] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750 [sys.catalog]: This master's current role is: LEADER
I20250114 02:54:00.241829 581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70: Bootstrap starting.
I20250114 02:54:00.242689 701 catalog_manager.cc:1348] Generated new cluster ID: f21d09cba41846a78a5fc0502d19012e
I20250114 02:54:00.242995 701 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250114 02:54:00.286891 684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b303b7df086340a0b5ad6373940ae750" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b303b7df086340a0b5ad6373940ae750" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 45367 } } peers { permanent_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } } peers { permanent_uuid: "87b3db09ac87436e9a60d6de5fedaff5" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } } }
I20250114 02:54:00.287659 684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [sys.catalog]: This master's current role is: FOLLOWER
I20250114 02:54:00.304479 581 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:00.307569 581 log.cc:826] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70: Log is configured to *not* fsync() on all Append() calls
I20250114 02:54:00.328522 581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70: No bootstrap required, opened a new log
I20250114 02:54:00.351265 701 catalog_manager.cc:1371] Generated new certificate authority record
I20250114 02:54:00.353389 701 catalog_manager.cc:1505] Loading token signing keys...
I20250114 02:54:00.381904 581 raft_consensus.cc:357] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b303b7df086340a0b5ad6373940ae750" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 45367 } } peers { permanent_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } } peers { permanent_uuid: "87b3db09ac87436e9a60d6de5fedaff5" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } }
I20250114 02:54:00.382812 581 raft_consensus.cc:383] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:00.383136 581 raft_consensus.cc:738] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 687d4ff6d3cb42cf90a965070b0d8a70, State: Initialized, Role: FOLLOWER
I20250114 02:54:00.384125 581 consensus_queue.cc:260] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b303b7df086340a0b5ad6373940ae750" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 45367 } } peers { permanent_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } } peers { permanent_uuid: "87b3db09ac87436e9a60d6de5fedaff5" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } }
I20250114 02:54:00.397209 710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b303b7df086340a0b5ad6373940ae750" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 45367 } } peers { permanent_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } } peers { permanent_uuid: "87b3db09ac87436e9a60d6de5fedaff5" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } } }
I20250114 02:54:00.398058 710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [sys.catalog]: This master's current role is: FOLLOWER
I20250114 02:54:00.401010 701 catalog_manager.cc:5899] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Generated new TSK 0
I20250114 02:54:00.404358 701 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250114 02:54:00.408104 581 sys_catalog.cc:564] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [sys.catalog]: configured and running, proceeding with master startup.
I20250114 02:54:00.408656 554 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:00.412453 556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b303b7df086340a0b5ad6373940ae750" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" is_pre_election: true
I20250114 02:54:00.414533 555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b303b7df086340a0b5ad6373940ae750" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "687d4ff6d3cb42cf90a965070b0d8a70"
I20250114 02:54:00.419981 554 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [term 1 FOLLOWER]: Refusing update from remote peer b303b7df086340a0b5ad6373940ae750: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 02:54:00.422426 672 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750 [LEADER]: Connected to new peer: Peer: permanent_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:54:00.444181 429 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrCUXNq/build/tsan/bin/kudu
/tmp/dist-test-taskrCUXNq/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.107.65:0
--local_ip_for_outbound_sockets=127.0.107.65
--webserver_interface=127.0.107.65
--webserver_port=0
--tserver_master_addrs=127.0.107.126:45367,127.0.107.125:46837,127.0.107.124:39105
--builtin_ntp_servers=127.0.107.84:40455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250114 02:54:00.503544 710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b303b7df086340a0b5ad6373940ae750. Latest consensus state: current_term: 1 leader_uuid: "b303b7df086340a0b5ad6373940ae750" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b303b7df086340a0b5ad6373940ae750" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 45367 } } peers { permanent_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } } peers { permanent_uuid: "87b3db09ac87436e9a60d6de5fedaff5" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } } }
I20250114 02:54:00.504467 710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [sys.catalog]: This master's current role is: FOLLOWER
I20250114 02:54:00.503561 722 sys_catalog.cc:455] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b303b7df086340a0b5ad6373940ae750" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b303b7df086340a0b5ad6373940ae750" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 45367 } } peers { permanent_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } } peers { permanent_uuid: "87b3db09ac87436e9a60d6de5fedaff5" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } } }
I20250114 02:54:00.505306 722 sys_catalog.cc:458] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [sys.catalog]: This master's current role is: FOLLOWER
I20250114 02:54:00.532119 727 catalog_manager.cc:1260] Loaded cluster ID: f21d09cba41846a78a5fc0502d19012e
I20250114 02:54:00.532505 727 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70: loading cluster ID for follower catalog manager: success
I20250114 02:54:00.542382 727 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70: acquiring CA information for follower catalog manager: success
I20250114 02:54:00.557103 727 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250114 02:54:00.818604 696 catalog_manager.cc:1260] Loaded cluster ID: f21d09cba41846a78a5fc0502d19012e
I20250114 02:54:00.818890 696 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5: loading cluster ID for follower catalog manager: success
I20250114 02:54:00.826186 696 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5: acquiring CA information for follower catalog manager: success
I20250114 02:54:00.841600 696 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250114 02:54:00.928380 722 raft_consensus.cc:491] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b303b7df086340a0b5ad6373940ae750)
I20250114 02:54:00.929096 722 raft_consensus.cc:513] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b303b7df086340a0b5ad6373940ae750" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 45367 } } peers { permanent_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } } peers { permanent_uuid: "87b3db09ac87436e9a60d6de5fedaff5" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } }
I20250114 02:54:00.933362 481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b303b7df086340a0b5ad6373940ae750" is_pre_election: true
I20250114 02:54:00.934044 722 leader_election.cc:290] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b303b7df086340a0b5ad6373940ae750 (127.0.107.126:45367), 87b3db09ac87436e9a60d6de5fedaff5 (127.0.107.124:39105)
I20250114 02:54:00.935194 635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "87b3db09ac87436e9a60d6de5fedaff5" is_pre_election: true
I20250114 02:54:00.936652 530 leader_election.cc:304] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 687d4ff6d3cb42cf90a965070b0d8a70; no voters: 87b3db09ac87436e9a60d6de5fedaff5, b303b7df086340a0b5ad6373940ae750
I20250114 02:54:00.938201 722 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:54:00.997130 672 sys_catalog.cc:455] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b303b7df086340a0b5ad6373940ae750" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b303b7df086340a0b5ad6373940ae750" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 45367 } } peers { permanent_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } } peers { permanent_uuid: "87b3db09ac87436e9a60d6de5fedaff5" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } } }
I20250114 02:54:00.997850 672 sys_catalog.cc:458] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750 [sys.catalog]: This master's current role is: LEADER
W20250114 02:54:00.998456 717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 02:54:00.999228 717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 02:54:01.000620 717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 02:54:01.059960 717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 02:54:01.061646 717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.107.65
I20250114 02:54:01.125175 717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.107.84:40455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.107.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.107.65
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.107.126:45367,127.0.107.125:46837,127.0.107.124:39105
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.107.65
--log_dir=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type FASTDEBUG
built by None at 14 Jan 2025 02:43:23 UTC on a633016ff5c9
build id 4168
TSAN enabled
I20250114 02:54:01.127137 717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 02:54:01.129300 717 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 02:54:01.173051 737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 02:54:01.520704 743 raft_consensus.cc:491] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b303b7df086340a0b5ad6373940ae750)
I20250114 02:54:01.521317 743 raft_consensus.cc:513] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b303b7df086340a0b5ad6373940ae750" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 45367 } } peers { permanent_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } } peers { permanent_uuid: "87b3db09ac87436e9a60d6de5fedaff5" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } }
I20250114 02:54:01.524016 481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b303b7df086340a0b5ad6373940ae750" is_pre_election: true
I20250114 02:54:01.528453 743 leader_election.cc:290] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b303b7df086340a0b5ad6373940ae750 (127.0.107.126:45367), 87b3db09ac87436e9a60d6de5fedaff5 (127.0.107.124:39105)
I20250114 02:54:01.528635 635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "87b3db09ac87436e9a60d6de5fedaff5" is_pre_election: true
I20250114 02:54:01.530068 530 leader_election.cc:304] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 687d4ff6d3cb42cf90a965070b0d8a70; no voters: 87b3db09ac87436e9a60d6de5fedaff5, b303b7df086340a0b5ad6373940ae750
I20250114 02:54:01.530787 743 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:54:02.133975 744 raft_consensus.cc:491] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b303b7df086340a0b5ad6373940ae750)
I20250114 02:54:02.136473 744 raft_consensus.cc:513] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b303b7df086340a0b5ad6373940ae750" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 45367 } } peers { permanent_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } } peers { permanent_uuid: "87b3db09ac87436e9a60d6de5fedaff5" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } }
I20250114 02:54:02.140081 481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "87b3db09ac87436e9a60d6de5fedaff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b303b7df086340a0b5ad6373940ae750" is_pre_election: true
I20250114 02:54:02.144101 744 leader_election.cc:290] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b303b7df086340a0b5ad6373940ae750 (127.0.107.126:45367), 687d4ff6d3cb42cf90a965070b0d8a70 (127.0.107.125:46837)
I20250114 02:54:02.144727 554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "87b3db09ac87436e9a60d6de5fedaff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" is_pre_election: true
I20250114 02:54:02.146409 609 leader_election.cc:304] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87b3db09ac87436e9a60d6de5fedaff5; no voters: 687d4ff6d3cb42cf90a965070b0d8a70, b303b7df086340a0b5ad6373940ae750
I20250114 02:54:02.149663 744 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250114 02:54:01.188668 741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 02:54:02.692461 739 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1522 milliseconds
I20250114 02:54:02.693017 717 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250114 02:54:01.173264 738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 02:54:02.697305 717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 02:54:02.707644 717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 02:54:02.709118 717 hybrid_clock.cc:648] HybridClock initialized: now 1736823242709052 us; error 79 us; skew 500 ppm
I20250114 02:54:02.710271 717 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250114 02:54:02.741012 717 webserver.cc:458] Webserver started at http://127.0.107.65:36635/ using document root <none> and password file <none>
I20250114 02:54:02.742509 717 fs_manager.cc:362] Metadata directory not provided
I20250114 02:54:02.742949 717 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 02:54:02.743568 717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 02:54:02.750960 717 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/data/instance:
uuid: "819f9e3412e54d44abb11d8b3ca877c2"
format_stamp: "Formatted at 2025-01-14 02:54:02 on dist-test-slave-36vg"
I20250114 02:54:02.753305 717 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/instance:
uuid: "819f9e3412e54d44abb11d8b3ca877c2"
format_stamp: "Formatted at 2025-01-14 02:54:02 on dist-test-slave-36vg"
I20250114 02:54:02.769129 717 fs_manager.cc:696] Time spent creating directory manager: real 0.015s user 0.005s sys 0.008s
I20250114 02:54:02.782639 749 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 02:54:02.783979 717 fs_manager.cc:730] Time spent opening block manager: real 0.010s user 0.004s sys 0.005s
I20250114 02:54:02.784880 717 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal
uuid: "819f9e3412e54d44abb11d8b3ca877c2"
format_stamp: "Formatted at 2025-01-14 02:54:02 on dist-test-slave-36vg"
I20250114 02:54:02.785420 717 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 02:54:02.919927 717 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 02:54:02.922194 717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 02:54:02.922866 717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 02:54:02.932415 717 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 02:54:02.947677 717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 02:54:02.948071 717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 02:54:02.948499 717 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 02:54:02.948781 717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 02:54:02.949167 756 raft_consensus.cc:491] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b303b7df086340a0b5ad6373940ae750)
I20250114 02:54:02.949728 756 raft_consensus.cc:513] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b303b7df086340a0b5ad6373940ae750" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 45367 } } peers { permanent_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } } peers { permanent_uuid: "87b3db09ac87436e9a60d6de5fedaff5" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } }
I20250114 02:54:02.952631 481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b303b7df086340a0b5ad6373940ae750" is_pre_election: true
I20250114 02:54:02.953789 635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "87b3db09ac87436e9a60d6de5fedaff5" is_pre_election: true
I20250114 02:54:02.954198 756 leader_election.cc:290] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b303b7df086340a0b5ad6373940ae750 (127.0.107.126:45367), 87b3db09ac87436e9a60d6de5fedaff5 (127.0.107.124:39105)
I20250114 02:54:02.955200 530 leader_election.cc:304] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 687d4ff6d3cb42cf90a965070b0d8a70; no voters: 87b3db09ac87436e9a60d6de5fedaff5, b303b7df086340a0b5ad6373940ae750
I20250114 02:54:02.956210 756 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:54:03.177130 717 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.65:39397
I20250114 02:54:03.177222 862 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.65:39397 every 8 connection(s)
I20250114 02:54:03.180543 717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/data/info.pb
I20250114 02:54:03.192524 429 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrCUXNq/build/tsan/bin/kudu as pid 717
I20250114 02:54:03.193156 429 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/instance
I20250114 02:54:03.217749 429 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrCUXNq/build/tsan/bin/kudu
/tmp/dist-test-taskrCUXNq/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.107.66:0
--local_ip_for_outbound_sockets=127.0.107.66
--webserver_interface=127.0.107.66
--webserver_port=0
--tserver_master_addrs=127.0.107.126:45367,127.0.107.125:46837,127.0.107.124:39105
--builtin_ntp_servers=127.0.107.84:40455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250114 02:54:03.238565 865 heartbeater.cc:346] Connected to a master server at 127.0.107.126:45367
I20250114 02:54:03.239081 865 heartbeater.cc:463] Registering TS with master...
I20250114 02:54:03.240361 865 heartbeater.cc:510] Master 127.0.107.126:45367 requested a full tablet report, sending...
I20250114 02:54:03.243124 471 ts_manager.cc:194] Registered new tserver with Master: 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:03.245695 471 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.107.65:49805
I20250114 02:54:03.256541 864 heartbeater.cc:346] Connected to a master server at 127.0.107.125:46837
I20250114 02:54:03.256947 864 heartbeater.cc:463] Registering TS with master...
I20250114 02:54:03.257638 864 heartbeater.cc:510] Master 127.0.107.125:46837 requested a full tablet report, sending...
I20250114 02:54:03.258610 863 heartbeater.cc:346] Connected to a master server at 127.0.107.124:39105
I20250114 02:54:03.258988 863 heartbeater.cc:463] Registering TS with master...
I20250114 02:54:03.259663 863 heartbeater.cc:510] Master 127.0.107.124:39105 requested a full tablet report, sending...
I20250114 02:54:03.262594 625 ts_manager.cc:194] Registered new tserver with Master: 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:03.262621 545 ts_manager.cc:194] Registered new tserver with Master: 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
W20250114 02:54:03.556872 873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 02:54:03.557391 873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 02:54:03.557953 873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 02:54:03.588249 873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 02:54:03.589126 873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.107.66
I20250114 02:54:03.622543 873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.107.84:40455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.107.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.107.66
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.107.126:45367,127.0.107.125:46837,127.0.107.124:39105
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.107.66
--log_dir=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type FASTDEBUG
built by None at 14 Jan 2025 02:43:23 UTC on a633016ff5c9
build id 4168
TSAN enabled
I20250114 02:54:03.623872 873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 02:54:03.625406 873 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 02:54:03.642946 879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 02:54:03.749902 865 heartbeater.cc:502] Master 127.0.107.126:45367 was elected leader, sending a full tablet report...
I20250114 02:54:03.881166 885 raft_consensus.cc:491] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b303b7df086340a0b5ad6373940ae750)
I20250114 02:54:03.881549 885 raft_consensus.cc:513] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b303b7df086340a0b5ad6373940ae750" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 45367 } } peers { permanent_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } } peers { permanent_uuid: "87b3db09ac87436e9a60d6de5fedaff5" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } }
I20250114 02:54:03.883024 885 leader_election.cc:290] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b303b7df086340a0b5ad6373940ae750 (127.0.107.126:45367), 87b3db09ac87436e9a60d6de5fedaff5 (127.0.107.124:39105)
I20250114 02:54:03.883788 481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b303b7df086340a0b5ad6373940ae750" is_pre_election: true
I20250114 02:54:03.884056 635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "87b3db09ac87436e9a60d6de5fedaff5" is_pre_election: true
I20250114 02:54:03.885913 530 leader_election.cc:304] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 687d4ff6d3cb42cf90a965070b0d8a70; no voters: 87b3db09ac87436e9a60d6de5fedaff5, b303b7df086340a0b5ad6373940ae750
I20250114 02:54:03.887028 885 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:54:03.965849 886 raft_consensus.cc:491] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b303b7df086340a0b5ad6373940ae750)
I20250114 02:54:03.966269 886 raft_consensus.cc:513] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b303b7df086340a0b5ad6373940ae750" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 45367 } } peers { permanent_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } } peers { permanent_uuid: "87b3db09ac87436e9a60d6de5fedaff5" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } }
I20250114 02:54:03.968120 886 leader_election.cc:290] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b303b7df086340a0b5ad6373940ae750 (127.0.107.126:45367), 687d4ff6d3cb42cf90a965070b0d8a70 (127.0.107.125:46837)
I20250114 02:54:03.969055 554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "87b3db09ac87436e9a60d6de5fedaff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" is_pre_election: true
I20250114 02:54:03.969122 481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "87b3db09ac87436e9a60d6de5fedaff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b303b7df086340a0b5ad6373940ae750" is_pre_election: true
I20250114 02:54:03.970431 609 leader_election.cc:304] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87b3db09ac87436e9a60d6de5fedaff5; no voters: 687d4ff6d3cb42cf90a965070b0d8a70, b303b7df086340a0b5ad6373940ae750
I20250114 02:54:03.970944 886 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:54:04.537024 888 raft_consensus.cc:491] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b303b7df086340a0b5ad6373940ae750)
I20250114 02:54:04.537479 888 raft_consensus.cc:513] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b303b7df086340a0b5ad6373940ae750" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 45367 } } peers { permanent_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } } peers { permanent_uuid: "87b3db09ac87436e9a60d6de5fedaff5" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } }
I20250114 02:54:04.538930 888 leader_election.cc:290] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b303b7df086340a0b5ad6373940ae750 (127.0.107.126:45367), 687d4ff6d3cb42cf90a965070b0d8a70 (127.0.107.125:46837)
I20250114 02:54:04.539804 481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "87b3db09ac87436e9a60d6de5fedaff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b303b7df086340a0b5ad6373940ae750" is_pre_election: true
I20250114 02:54:04.540264 554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "87b3db09ac87436e9a60d6de5fedaff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" is_pre_election: true
I20250114 02:54:04.542021 609 leader_election.cc:304] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87b3db09ac87436e9a60d6de5fedaff5; no voters: 687d4ff6d3cb42cf90a965070b0d8a70, b303b7df086340a0b5ad6373940ae750
I20250114 02:54:04.543093 888 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250114 02:54:03.643606 882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 02:54:03.645407 873 server_base.cc:1034] running on GCE node
W20250114 02:54:03.645069 880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 02:54:04.805697 873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 02:54:04.807921 873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 02:54:04.809309 873 hybrid_clock.cc:648] HybridClock initialized: now 1736823244809275 us; error 44 us; skew 500 ppm
I20250114 02:54:04.810177 873 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250114 02:54:04.816414 873 webserver.cc:458] Webserver started at http://127.0.107.66:35021/ using document root <none> and password file <none>
I20250114 02:54:04.817278 873 fs_manager.cc:362] Metadata directory not provided
I20250114 02:54:04.817464 873 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 02:54:04.817862 873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 02:54:04.822541 873 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/data/instance:
uuid: "525198593c8e47afaee54e74267b87e3"
format_stamp: "Formatted at 2025-01-14 02:54:04 on dist-test-slave-36vg"
I20250114 02:54:04.823527 873 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/instance:
uuid: "525198593c8e47afaee54e74267b87e3"
format_stamp: "Formatted at 2025-01-14 02:54:04 on dist-test-slave-36vg"
I20250114 02:54:04.830729 873 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20250114 02:54:04.836274 894 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 02:54:04.837430 873 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250114 02:54:04.837708 873 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal
uuid: "525198593c8e47afaee54e74267b87e3"
format_stamp: "Formatted at 2025-01-14 02:54:04 on dist-test-slave-36vg"
I20250114 02:54:04.838021 873 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 02:54:04.889647 873 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 02:54:04.891156 873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 02:54:04.891572 873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 02:54:04.893981 873 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 02:54:04.897879 873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 02:54:04.898113 873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 02:54:04.898353 873 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 02:54:04.898510 873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 02:54:05.044023 873 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.66:44037
I20250114 02:54:05.044092 1006 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.66:44037 every 8 connection(s)
I20250114 02:54:05.047475 873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/data/info.pb
I20250114 02:54:05.054051 429 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrCUXNq/build/tsan/bin/kudu as pid 873
I20250114 02:54:05.055116 429 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/instance
I20250114 02:54:05.069679 1012 raft_consensus.cc:491] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b303b7df086340a0b5ad6373940ae750)
I20250114 02:54:05.071098 1012 raft_consensus.cc:513] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b303b7df086340a0b5ad6373940ae750" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 45367 } } peers { permanent_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } } peers { permanent_uuid: "87b3db09ac87436e9a60d6de5fedaff5" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } }
I20250114 02:54:05.073679 1012 leader_election.cc:290] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b303b7df086340a0b5ad6373940ae750 (127.0.107.126:45367), 687d4ff6d3cb42cf90a965070b0d8a70 (127.0.107.125:46837)
I20250114 02:54:05.075132 481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "87b3db09ac87436e9a60d6de5fedaff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b303b7df086340a0b5ad6373940ae750" is_pre_election: true
I20250114 02:54:05.073264 429 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrCUXNq/build/tsan/bin/kudu
/tmp/dist-test-taskrCUXNq/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.107.67:0
--local_ip_for_outbound_sockets=127.0.107.67
--webserver_interface=127.0.107.67
--webserver_port=0
--tserver_master_addrs=127.0.107.126:45367,127.0.107.125:46837,127.0.107.124:39105
--builtin_ntp_servers=127.0.107.84:40455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250114 02:54:05.093518 554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "87b3db09ac87436e9a60d6de5fedaff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" is_pre_election: true
I20250114 02:54:05.095199 609 leader_election.cc:304] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87b3db09ac87436e9a60d6de5fedaff5; no voters: 687d4ff6d3cb42cf90a965070b0d8a70, b303b7df086340a0b5ad6373940ae750
I20250114 02:54:05.096110 1012 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:54:05.113940 1008 heartbeater.cc:346] Connected to a master server at 127.0.107.125:46837
I20250114 02:54:05.114387 1008 heartbeater.cc:463] Registering TS with master...
I20250114 02:54:05.114428 1007 heartbeater.cc:346] Connected to a master server at 127.0.107.124:39105
I20250114 02:54:05.114509 1009 heartbeater.cc:346] Connected to a master server at 127.0.107.126:45367
I20250114 02:54:05.114840 1007 heartbeater.cc:463] Registering TS with master...
I20250114 02:54:05.114864 1009 heartbeater.cc:463] Registering TS with master...
I20250114 02:54:05.115852 1008 heartbeater.cc:510] Master 127.0.107.125:46837 requested a full tablet report, sending...
I20250114 02:54:05.116102 1009 heartbeater.cc:510] Master 127.0.107.126:45367 requested a full tablet report, sending...
I20250114 02:54:05.116065 1007 heartbeater.cc:510] Master 127.0.107.124:39105 requested a full tablet report, sending...
I20250114 02:54:05.119228 471 ts_manager.cc:194] Registered new tserver with Master: 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:05.119519 625 ts_manager.cc:194] Registered new tserver with Master: 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:05.121912 471 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.107.66:50663
I20250114 02:54:05.124760 545 ts_manager.cc:194] Registered new tserver with Master: 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
W20250114 02:54:05.398867 1017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 02:54:05.399385 1017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 02:54:05.399947 1017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 02:54:05.429941 1017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 02:54:05.430882 1017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.107.67
I20250114 02:54:05.451648 1023 raft_consensus.cc:491] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b303b7df086340a0b5ad6373940ae750)
I20250114 02:54:05.452014 1023 raft_consensus.cc:513] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b303b7df086340a0b5ad6373940ae750" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 45367 } } peers { permanent_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } } peers { permanent_uuid: "87b3db09ac87436e9a60d6de5fedaff5" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } }
I20250114 02:54:05.453378 1023 leader_election.cc:290] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b303b7df086340a0b5ad6373940ae750 (127.0.107.126:45367), 87b3db09ac87436e9a60d6de5fedaff5 (127.0.107.124:39105)
I20250114 02:54:05.454363 481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b303b7df086340a0b5ad6373940ae750" is_pre_election: true
I20250114 02:54:05.454476 635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "87b3db09ac87436e9a60d6de5fedaff5" is_pre_election: true
I20250114 02:54:05.455963 532 leader_election.cc:304] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 687d4ff6d3cb42cf90a965070b0d8a70; no voters: 87b3db09ac87436e9a60d6de5fedaff5, b303b7df086340a0b5ad6373940ae750
I20250114 02:54:05.456542 1023 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:54:05.466116 1017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.107.84:40455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.107.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.107.67
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.107.126:45367,127.0.107.125:46837,127.0.107.124:39105
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.107.67
--log_dir=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type FASTDEBUG
built by None at 14 Jan 2025 02:43:23 UTC on a633016ff5c9
build id 4168
TSAN enabled
I20250114 02:54:05.467799 1017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 02:54:05.469376 1017 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 02:54:05.488644 1026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 02:54:05.627334 1009 heartbeater.cc:502] Master 127.0.107.126:45367 was elected leader, sending a full tablet report...
I20250114 02:54:05.968348 1031 raft_consensus.cc:491] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b303b7df086340a0b5ad6373940ae750)
I20250114 02:54:05.968732 1031 raft_consensus.cc:513] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b303b7df086340a0b5ad6373940ae750" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 45367 } } peers { permanent_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } } peers { permanent_uuid: "87b3db09ac87436e9a60d6de5fedaff5" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } }
I20250114 02:54:05.970183 1031 leader_election.cc:290] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b303b7df086340a0b5ad6373940ae750 (127.0.107.126:45367), 87b3db09ac87436e9a60d6de5fedaff5 (127.0.107.124:39105)
I20250114 02:54:05.971199 481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b303b7df086340a0b5ad6373940ae750" is_pre_election: true
I20250114 02:54:05.971211 635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "87b3db09ac87436e9a60d6de5fedaff5" is_pre_election: true
I20250114 02:54:05.973469 532 leader_election.cc:304] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 687d4ff6d3cb42cf90a965070b0d8a70; no voters: 87b3db09ac87436e9a60d6de5fedaff5, b303b7df086340a0b5ad6373940ae750
I20250114 02:54:05.974514 1031 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:54:06.505370 1033 raft_consensus.cc:491] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b303b7df086340a0b5ad6373940ae750)
I20250114 02:54:06.506067 1033 raft_consensus.cc:513] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b303b7df086340a0b5ad6373940ae750" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 45367 } } peers { permanent_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } } peers { permanent_uuid: "87b3db09ac87436e9a60d6de5fedaff5" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } }
I20250114 02:54:06.508316 1033 leader_election.cc:290] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b303b7df086340a0b5ad6373940ae750 (127.0.107.126:45367), 87b3db09ac87436e9a60d6de5fedaff5 (127.0.107.124:39105)
I20250114 02:54:06.509212 481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b303b7df086340a0b5ad6373940ae750" is_pre_election: true
I20250114 02:54:06.509917 635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "87b3db09ac87436e9a60d6de5fedaff5" is_pre_election: true
I20250114 02:54:06.511374 530 leader_election.cc:304] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 687d4ff6d3cb42cf90a965070b0d8a70; no voters: 87b3db09ac87436e9a60d6de5fedaff5, b303b7df086340a0b5ad6373940ae750
I20250114 02:54:06.512292 1033 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250114 02:54:05.489104 1025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 02:54:05.489741 1017 server_base.cc:1034] running on GCE node
W20250114 02:54:05.490628 1028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 02:54:06.669552 1017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 02:54:06.671701 1017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 02:54:06.673023 1017 hybrid_clock.cc:648] HybridClock initialized: now 1736823246672980 us; error 55 us; skew 500 ppm
I20250114 02:54:06.673893 1017 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250114 02:54:06.679987 1017 webserver.cc:458] Webserver started at http://127.0.107.67:34525/ using document root <none> and password file <none>
I20250114 02:54:06.680948 1017 fs_manager.cc:362] Metadata directory not provided
I20250114 02:54:06.681164 1017 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 02:54:06.681586 1017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 02:54:06.686581 1017 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/data/instance:
uuid: "90d6c2c58bb54e0eafb365625ac33871"
format_stamp: "Formatted at 2025-01-14 02:54:06 on dist-test-slave-36vg"
I20250114 02:54:06.687692 1017 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/instance:
uuid: "90d6c2c58bb54e0eafb365625ac33871"
format_stamp: "Formatted at 2025-01-14 02:54:06 on dist-test-slave-36vg"
I20250114 02:54:06.694881 1017 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20250114 02:54:06.700886 1038 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 02:54:06.702013 1017 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250114 02:54:06.702306 1017 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal
uuid: "90d6c2c58bb54e0eafb365625ac33871"
format_stamp: "Formatted at 2025-01-14 02:54:06 on dist-test-slave-36vg"
I20250114 02:54:06.702620 1017 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 02:54:06.751083 1017 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 02:54:06.752548 1017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 02:54:06.752987 1017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 02:54:06.755362 1017 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 02:54:06.759302 1017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 02:54:06.759485 1017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 02:54:06.759676 1017 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 02:54:06.759825 1017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 02:54:06.921672 1017 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.67:45749
I20250114 02:54:06.921770 1150 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.67:45749 every 8 connection(s)
I20250114 02:54:06.924120 1017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/data/info.pb
I20250114 02:54:06.925858 429 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrCUXNq/build/tsan/bin/kudu as pid 1017
I20250114 02:54:06.926455 429 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/instance
I20250114 02:54:06.976970 1151 heartbeater.cc:346] Connected to a master server at 127.0.107.124:39105
I20250114 02:54:06.977383 1151 heartbeater.cc:463] Registering TS with master...
I20250114 02:54:06.978600 1151 heartbeater.cc:510] Master 127.0.107.124:39105 requested a full tablet report, sending...
I20250114 02:54:06.981812 625 ts_manager.cc:194] Registered new tserver with Master: 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:06.984941 1153 heartbeater.cc:346] Connected to a master server at 127.0.107.126:45367
I20250114 02:54:06.985198 1152 heartbeater.cc:346] Connected to a master server at 127.0.107.125:46837
I20250114 02:54:06.985430 1153 heartbeater.cc:463] Registering TS with master...
I20250114 02:54:06.985471 1152 heartbeater.cc:463] Registering TS with master...
I20250114 02:54:06.986724 1153 heartbeater.cc:510] Master 127.0.107.126:45367 requested a full tablet report, sending...
I20250114 02:54:06.986753 1152 heartbeater.cc:510] Master 127.0.107.125:46837 requested a full tablet report, sending...
I20250114 02:54:06.988689 471 ts_manager.cc:194] Registered new tserver with Master: 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:06.989574 545 ts_manager.cc:194] Registered new tserver with Master: 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:06.990322 471 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.107.67:34825
I20250114 02:54:06.993662 429 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250114 02:54:07.048152 471 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_6c28a519f1744def94f815c78e2ae532"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:07.052196 470 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_c6918b535d4c4489b30bfa370adcb0c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:07.052986 471 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6c28a519f1744def94f815c78e2ae532 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250114 02:54:07.054347 470 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c6918b535d4c4489b30bfa370adcb0c5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:07.067062 469 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_100975dd801240c694012379b37f5f1b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:07.069356 469 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_100975dd801240c694012379b37f5f1b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:07.077479 468 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_e750bf1faacf4043b6717ef4c0c20d89"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:07.079216 468 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e750bf1faacf4043b6717ef4c0c20d89 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:07.199563 470 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_c6918b535d4c4489b30bfa370adcb0c5" } new_table_name: "default.Table_fc3d1fa0eaac41ae9dbc89a9e4340f8f" modify_external_catalogs: true
I20250114 02:54:07.205018 469 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_6c28a519f1744def94f815c78e2ae532" } modify_external_catalogs: true
I20250114 02:54:07.205593 469 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_6c28a519f1744def94f815c78e2ae532" } modify_external_catalogs: true
I20250114 02:54:07.213201 468 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_e750bf1faacf4043b6717ef4c0c20d89" } modify_external_catalogs: true
I20250114 02:54:07.213717 468 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_e750bf1faacf4043b6717ef4c0c20d89" } modify_external_catalogs: true
I20250114 02:54:07.219300 471 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_100975dd801240c694012379b37f5f1b" } new_table_name: "default.Table_3fc4e989c384455795b08a5f0559551f" modify_external_catalogs: true
I20250114 02:54:07.230065 429 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrCUXNq/build/tsan/bin/kudu with pid 583
W20250114 02:54:07.245976 470 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a1cf92e5275b454ca893b12f2d90d25b has no consensus state
W20250114 02:54:07.252993 471 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fa47e448d3f746048afccbd2f0fe3555 has no consensus state
I20250114 02:54:07.255275 468 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet eec379ac77844af191de904cdc061191
I20250114 02:54:07.267863 469 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 743b618711e747e090071e14af6ff7d4
W20250114 02:54:07.270921 455 connection.cc:537] client connection to 127.0.107.124:39105 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250114 02:54:07.271199 455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 12 similar messages]
W20250114 02:54:07.283624 455 consensus_peers.cc:487] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750 -> Peer 87b3db09ac87436e9a60d6de5fedaff5 (127.0.107.124:39105): Couldn't send request to peer 87b3db09ac87436e9a60d6de5fedaff5. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.124:39105: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250114 02:54:07.292299 455 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet fa47e448d3f746048afccbd2f0fe3555 (table default.Table_3fc4e989c384455795b08a5f0559551f [id=d2910ff0aac44e5e8f0d1ee6918255e9], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet fa47e448d3f746048afccbd2f0fe3555 consensus state has no leader
W20250114 02:54:07.306721 863 heartbeater.cc:643] Failed to heartbeat to 127.0.107.124:39105 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.107.124:39105: connect: Connection refused (error 111)
I20250114 02:54:07.314728 942 tablet_service.cc:1467] Processing CreateTablet for tablet fa47e448d3f746048afccbd2f0fe3555 (DEFAULT_TABLE table=default.Table_3fc4e989c384455795b08a5f0559551f [id=d2910ff0aac44e5e8f0d1ee6918255e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:07.314787 941 tablet_service.cc:1467] Processing CreateTablet for tablet a1cf92e5275b454ca893b12f2d90d25b (DEFAULT_TABLE table=default.Table_fc3d1fa0eaac41ae9dbc89a9e4340f8f [id=9ec503e449fb4a2a96bbb77237c2fb4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:07.316713 942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa47e448d3f746048afccbd2f0fe3555. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:07.317520 941 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1cf92e5275b454ca893b12f2d90d25b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:07.322144 797 tablet_service.cc:1467] Processing CreateTablet for tablet a1cf92e5275b454ca893b12f2d90d25b (DEFAULT_TABLE table=default.Table_fc3d1fa0eaac41ae9dbc89a9e4340f8f [id=9ec503e449fb4a2a96bbb77237c2fb4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:07.322185 798 tablet_service.cc:1467] Processing CreateTablet for tablet fa47e448d3f746048afccbd2f0fe3555 (DEFAULT_TABLE table=default.Table_3fc4e989c384455795b08a5f0559551f [id=d2910ff0aac44e5e8f0d1ee6918255e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:07.324374 797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1cf92e5275b454ca893b12f2d90d25b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:07.332614 798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa47e448d3f746048afccbd2f0fe3555. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:07.346140 1085 tablet_service.cc:1467] Processing CreateTablet for tablet a1cf92e5275b454ca893b12f2d90d25b (DEFAULT_TABLE table=default.Table_fc3d1fa0eaac41ae9dbc89a9e4340f8f [id=9ec503e449fb4a2a96bbb77237c2fb4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:07.347939 1085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1cf92e5275b454ca893b12f2d90d25b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:07.346140 1086 tablet_service.cc:1467] Processing CreateTablet for tablet fa47e448d3f746048afccbd2f0fe3555 (DEFAULT_TABLE table=default.Table_3fc4e989c384455795b08a5f0559551f [id=d2910ff0aac44e5e8f0d1ee6918255e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:07.353006 1086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa47e448d3f746048afccbd2f0fe3555. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:07.380812 469 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_3c0778147f834283bfa9492ba941b12e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:07.381562 1205 tablet_bootstrap.cc:492] T a1cf92e5275b454ca893b12f2d90d25b P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
W20250114 02:54:07.382508 469 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3c0778147f834283bfa9492ba941b12e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:07.386078 468 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_f4654530e4b34b768a0b9691d76a0532"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:07.393090 468 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f4654530e4b34b768a0b9691d76a0532 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:07.396559 471 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_8f9e13c0f3be4c889373faa3988f62d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:07.398610 471 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8f9e13c0f3be4c889373faa3988f62d7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:07.405741 1206 tablet_bootstrap.cc:492] T fa47e448d3f746048afccbd2f0fe3555 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:07.415216 470 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_e7d101746de1409ca560602b343f7eef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:07.417400 470 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e7d101746de1409ca560602b343f7eef in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:07.413018 1206 tablet_bootstrap.cc:654] T fa47e448d3f746048afccbd2f0fe3555 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:07.426679 1206 log.cc:826] T fa47e448d3f746048afccbd2f0fe3555 P 819f9e3412e54d44abb11d8b3ca877c2: Log is configured to *not* fsync() on all Append() calls
I20250114 02:54:07.431051 1205 tablet_bootstrap.cc:654] T a1cf92e5275b454ca893b12f2d90d25b P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:07.433876 1205 log.cc:826] T a1cf92e5275b454ca893b12f2d90d25b P 525198593c8e47afaee54e74267b87e3: Log is configured to *not* fsync() on all Append() calls
I20250114 02:54:07.446856 1205 tablet_bootstrap.cc:492] T a1cf92e5275b454ca893b12f2d90d25b P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:07.447399 1205 ts_tablet_manager.cc:1397] T a1cf92e5275b454ca893b12f2d90d25b P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.066s user 0.009s sys 0.005s
I20250114 02:54:07.456542 469 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_fc3d1fa0eaac41ae9dbc89a9e4340f8f" } new_table_name: "default.Table_64a15911f1d14ca882ff3b30ef9b3df9" modify_external_catalogs: true
I20250114 02:54:07.457430 468 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_8f9e13c0f3be4c889373faa3988f62d7" } new_table_name: "default.Table_dc1f5f3f0ee646889739cfb95dbcf2bb" modify_external_catalogs: true
I20250114 02:54:07.458765 1206 tablet_bootstrap.cc:492] T fa47e448d3f746048afccbd2f0fe3555 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:07.458775 471 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_3fc4e989c384455795b08a5f0559551f" } modify_external_catalogs: true
I20250114 02:54:07.459278 1206 ts_tablet_manager.cc:1397] T fa47e448d3f746048afccbd2f0fe3555 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.055s user 0.016s sys 0.006s
I20250114 02:54:07.459277 471 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_3fc4e989c384455795b08a5f0559551f" } modify_external_catalogs: true
I20250114 02:54:07.481099 1207 tablet_bootstrap.cc:492] T fa47e448d3f746048afccbd2f0fe3555 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:07.482862 467 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_3c0778147f834283bfa9492ba941b12e" } modify_external_catalogs: true
I20250114 02:54:07.483716 467 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_3c0778147f834283bfa9492ba941b12e" } modify_external_catalogs: true
I20250114 02:54:07.506959 1205 raft_consensus.cc:357] T a1cf92e5275b454ca893b12f2d90d25b P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:07.508074 1205 raft_consensus.cc:383] T a1cf92e5275b454ca893b12f2d90d25b P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:07.508422 1205 raft_consensus.cc:738] T a1cf92e5275b454ca893b12f2d90d25b P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:07.509543 1205 consensus_queue.cc:260] T a1cf92e5275b454ca893b12f2d90d25b P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:07.516777 1206 raft_consensus.cc:357] T fa47e448d3f746048afccbd2f0fe3555 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:07.517760 1206 raft_consensus.cc:383] T fa47e448d3f746048afccbd2f0fe3555 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:07.518039 1206 raft_consensus.cc:738] T fa47e448d3f746048afccbd2f0fe3555 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:07.519166 1206 consensus_queue.cc:260] T fa47e448d3f746048afccbd2f0fe3555 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:07.523833 1205 ts_tablet_manager.cc:1428] T a1cf92e5275b454ca893b12f2d90d25b P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.076s user 0.041s sys 0.008s
I20250114 02:54:07.527946 798 tablet_service.cc:1467] Processing CreateTablet for tablet 57eabef786d748e0ab8ef03b29819514 (DEFAULT_TABLE table=default.Table_3c0778147f834283bfa9492ba941b12e [id=5b896aa662344869896925561689b5ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:07.529258 798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57eabef786d748e0ab8ef03b29819514. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:07.530762 1206 ts_tablet_manager.cc:1428] T fa47e448d3f746048afccbd2f0fe3555 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.071s user 0.038s sys 0.005s
I20250114 02:54:07.531612 1206 tablet_bootstrap.cc:492] T a1cf92e5275b454ca893b12f2d90d25b P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:07.537407 1206 tablet_bootstrap.cc:654] T a1cf92e5275b454ca893b12f2d90d25b P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
W20250114 02:54:07.539273 469 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a1cf92e5275b454ca893b12f2d90d25b consensus state has no leader
I20250114 02:54:07.532514 1212 raft_consensus.cc:491] T fa47e448d3f746048afccbd2f0fe3555 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:07.540592 1212 raft_consensus.cc:513] T fa47e448d3f746048afccbd2f0fe3555 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:07.493872 1153 heartbeater.cc:502] Master 127.0.107.126:45367 was elected leader, sending a full tablet report...
I20250114 02:54:07.543130 1086 tablet_service.cc:1467] Processing CreateTablet for tablet 57eabef786d748e0ab8ef03b29819514 (DEFAULT_TABLE table=default.Table_3c0778147f834283bfa9492ba941b12e [id=5b896aa662344869896925561689b5ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:07.544351 1086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57eabef786d748e0ab8ef03b29819514. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:07.546671 1212 leader_election.cc:290] T fa47e448d3f746048afccbd2f0fe3555 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
W20250114 02:54:07.562404 1151 heartbeater.cc:643] Failed to heartbeat to 127.0.107.124:39105 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.107.124:39105: connect: Connection refused (error 111)
I20250114 02:54:07.565449 1205 tablet_bootstrap.cc:492] T fa47e448d3f746048afccbd2f0fe3555 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:07.568964 466 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet a1cf92e5275b454ca893b12f2d90d25b (table default.Table_64a15911f1d14ca882ff3b30ef9b3df9 [id=9ec503e449fb4a2a96bbb77237c2fb4f]). Expected version 2 got 0
I20250114 02:54:07.571646 1205 tablet_bootstrap.cc:654] T fa47e448d3f746048afccbd2f0fe3555 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:07.574362 1207 tablet_bootstrap.cc:654] T fa47e448d3f746048afccbd2f0fe3555 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:07.576946 941 tablet_service.cc:1467] Processing CreateTablet for tablet 57eabef786d748e0ab8ef03b29819514 (DEFAULT_TABLE table=default.Table_3c0778147f834283bfa9492ba941b12e [id=5b896aa662344869896925561689b5ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:07.578279 941 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57eabef786d748e0ab8ef03b29819514. 1 dirs total, 0 dirs full, 0 dirs failed
W20250114 02:54:07.596626 1013 tablet.cc:2367] T a1cf92e5275b454ca893b12f2d90d25b P 525198593c8e47afaee54e74267b87e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250114 02:54:07.599969 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa47e448d3f746048afccbd2f0fe3555" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:07.600154 1207 log.cc:826] T fa47e448d3f746048afccbd2f0fe3555 P 90d6c2c58bb54e0eafb365625ac33871: Log is configured to *not* fsync() on all Append() calls
W20250114 02:54:07.609131 751 leader_election.cc:343] T fa47e448d3f746048afccbd2f0fe3555 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:07.598074 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa47e448d3f746048afccbd2f0fe3555" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
W20250114 02:54:07.612761 753 leader_election.cc:343] T fa47e448d3f746048afccbd2f0fe3555 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:07.613155 753 leader_election.cc:304] T fa47e448d3f746048afccbd2f0fe3555 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:07.614296 1212 raft_consensus.cc:2743] T fa47e448d3f746048afccbd2f0fe3555 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:07.619294 1206 tablet_bootstrap.cc:492] T a1cf92e5275b454ca893b12f2d90d25b P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
W20250114 02:54:07.619424 1007 heartbeater.cc:643] Failed to heartbeat to 127.0.107.124:39105 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.107.124:39105: connect: Connection refused (error 111)
I20250114 02:54:07.619700 1206 ts_tablet_manager.cc:1397] T a1cf92e5275b454ca893b12f2d90d25b P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.088s user 0.008s sys 0.015s
I20250114 02:54:07.624191 466 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_9172ebaa2b004d3083584cd19c9a2617"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:07.624318 1206 raft_consensus.cc:357] T a1cf92e5275b454ca893b12f2d90d25b P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:07.625346 1206 raft_consensus.cc:383] T a1cf92e5275b454ca893b12f2d90d25b P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:07.625751 1206 raft_consensus.cc:738] T a1cf92e5275b454ca893b12f2d90d25b P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
W20250114 02:54:07.625875 466 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9172ebaa2b004d3083584cd19c9a2617 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:07.626742 1206 consensus_queue.cc:260] T a1cf92e5275b454ca893b12f2d90d25b P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:07.629357 1206 ts_tablet_manager.cc:1428] T a1cf92e5275b454ca893b12f2d90d25b P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20250114 02:54:07.630422 1206 tablet_bootstrap.cc:492] T 57eabef786d748e0ab8ef03b29819514 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
W20250114 02:54:07.633574 468 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 96698ba107b0446cb4e847ad7a90783c has no consensus state
I20250114 02:54:07.636828 469 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet a1cf92e5275b454ca893b12f2d90d25b (table default.Table_64a15911f1d14ca882ff3b30ef9b3df9 [id=9ec503e449fb4a2a96bbb77237c2fb4f]). Expected version 2 got 0
I20250114 02:54:07.638365 471 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet fa47e448d3f746048afccbd2f0fe3555
I20250114 02:54:07.638633 1206 tablet_bootstrap.cc:654] T 57eabef786d748e0ab8ef03b29819514 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:07.641392 798 tablet_service.cc:1514] Processing DeleteTablet for tablet fa47e448d3f746048afccbd2f0fe3555 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:07 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:07.642046 1086 tablet_service.cc:1514] Processing DeleteTablet for tablet fa47e448d3f746048afccbd2f0fe3555 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:07 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:07.645614 467 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet 57eabef786d748e0ab8ef03b29819514
I20250114 02:54:07.653654 797 tablet_service.cc:1514] Processing DeleteTablet for tablet 57eabef786d748e0ab8ef03b29819514 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:07 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:07.654286 1085 tablet_service.cc:1514] Processing DeleteTablet for tablet 57eabef786d748e0ab8ef03b29819514 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:07 UTC) from {username='slave'} at 127.0.0.1:53876
W20250114 02:54:07.658576 458 catalog_manager.cc:4852] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet fa47e448d3f746048afccbd2f0fe3555 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fa47e448d3f746048afccbd2f0fe3555 already in progress: creating tablet
I20250114 02:54:07.659518 1222 tablet_replica.cc:331] T fa47e448d3f746048afccbd2f0fe3555 P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
W20250114 02:54:07.660776 458 catalog_manager.cc:4852] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 57eabef786d748e0ab8ef03b29819514 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 57eabef786d748e0ab8ef03b29819514 already in progress: creating tablet
I20250114 02:54:07.662079 1222 raft_consensus.cc:2238] T fa47e448d3f746048afccbd2f0fe3555 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:07.662971 1222 raft_consensus.cc:2267] T fa47e448d3f746048afccbd2f0fe3555 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:07.674713 1222 ts_tablet_manager.cc:1905] T fa47e448d3f746048afccbd2f0fe3555 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:07.678459 1211 raft_consensus.cc:491] T a1cf92e5275b454ca893b12f2d90d25b P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:07.679224 1211 raft_consensus.cc:513] T a1cf92e5275b454ca893b12f2d90d25b P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:07.682605 1206 tablet_bootstrap.cc:492] T 57eabef786d748e0ab8ef03b29819514 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:07.682998 1206 ts_tablet_manager.cc:1397] T 57eabef786d748e0ab8ef03b29819514 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.053s user 0.016s sys 0.026s
I20250114 02:54:07.686041 1206 raft_consensus.cc:357] T 57eabef786d748e0ab8ef03b29819514 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:07.686748 1206 raft_consensus.cc:383] T 57eabef786d748e0ab8ef03b29819514 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:07.687038 1206 raft_consensus.cc:738] T 57eabef786d748e0ab8ef03b29819514 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:07.687678 1206 consensus_queue.cc:260] T 57eabef786d748e0ab8ef03b29819514 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:07.691298 1206 ts_tablet_manager.cc:1428] T 57eabef786d748e0ab8ef03b29819514 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20250114 02:54:07.700498 1211 leader_election.cc:290] T a1cf92e5275b454ca893b12f2d90d25b P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:07.704854 1212 raft_consensus.cc:491] T 57eabef786d748e0ab8ef03b29819514 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:07.705448 1212 raft_consensus.cc:513] T 57eabef786d748e0ab8ef03b29819514 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:07.706292 1222 ts_tablet_manager.cc:1918] T fa47e448d3f746048afccbd2f0fe3555 P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:07.706849 1222 log.cc:1198] T fa47e448d3f746048afccbd2f0fe3555 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/fa47e448d3f746048afccbd2f0fe3555
I20250114 02:54:07.707769 1222 ts_tablet_manager.cc:1939] T fa47e448d3f746048afccbd2f0fe3555 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:07.708891 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57eabef786d748e0ab8ef03b29819514" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
W20250114 02:54:07.710067 753 leader_election.cc:343] T 57eabef786d748e0ab8ef03b29819514 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:07.708020 1212 leader_election.cc:290] T 57eabef786d748e0ab8ef03b29819514 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
W20250114 02:54:07.710961 868 tablet.cc:2367] T 57eabef786d748e0ab8ef03b29819514 P 819f9e3412e54d44abb11d8b3ca877c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250114 02:54:07.712687 1205 tablet_bootstrap.cc:492] T fa47e448d3f746048afccbd2f0fe3555 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:07.712888 1222 tablet_replica.cc:331] T 57eabef786d748e0ab8ef03b29819514 P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:07.713006 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet fa47e448d3f746048afccbd2f0fe3555 (table default.Table_3fc4e989c384455795b08a5f0559551f [id=d2910ff0aac44e5e8f0d1ee6918255e9]) successfully deleted
I20250114 02:54:07.713279 1205 ts_tablet_manager.cc:1397] T fa47e448d3f746048afccbd2f0fe3555 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.155s user 0.000s sys 0.012s
I20250114 02:54:07.713495 1222 raft_consensus.cc:2238] T 57eabef786d748e0ab8ef03b29819514 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:07.713908 1222 raft_consensus.cc:2267] T 57eabef786d748e0ab8ef03b29819514 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:07.716779 1205 raft_consensus.cc:357] T fa47e448d3f746048afccbd2f0fe3555 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:07.717778 1205 raft_consensus.cc:383] T fa47e448d3f746048afccbd2f0fe3555 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:07.718140 1205 raft_consensus.cc:738] T fa47e448d3f746048afccbd2f0fe3555 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:07.720281 941 tablet_service.cc:1514] Processing DeleteTablet for tablet fa47e448d3f746048afccbd2f0fe3555 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:07 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:07.721915 942 tablet_service.cc:1514] Processing DeleteTablet for tablet 57eabef786d748e0ab8ef03b29819514 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:07 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:07.724131 1222 ts_tablet_manager.cc:1905] T 57eabef786d748e0ab8ef03b29819514 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:07.719048 1205 consensus_queue.cc:260] T fa47e448d3f746048afccbd2f0fe3555 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:07.725802 1207 tablet_bootstrap.cc:492] T fa47e448d3f746048afccbd2f0fe3555 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:07.726325 1207 ts_tablet_manager.cc:1397] T fa47e448d3f746048afccbd2f0fe3555 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.246s user 0.009s sys 0.006s
I20250114 02:54:07.729414 1205 ts_tablet_manager.cc:1428] T fa47e448d3f746048afccbd2f0fe3555 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.016s user 0.006s sys 0.003s
I20250114 02:54:07.730250 1205 tablet_bootstrap.cc:492] T 57eabef786d748e0ab8ef03b29819514 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:07.741919 1225 tablet_replica.cc:331] T fa47e448d3f746048afccbd2f0fe3555 P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:07.742769 1225 raft_consensus.cc:2238] T fa47e448d3f746048afccbd2f0fe3555 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:07.743520 1225 raft_consensus.cc:2267] T fa47e448d3f746048afccbd2f0fe3555 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:07.745038 1205 tablet_bootstrap.cc:654] T 57eabef786d748e0ab8ef03b29819514 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:07.744659 470 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_c6640103d84c48beb554d7509b246b27"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:07.746304 470 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c6640103d84c48beb554d7509b246b27 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:07.748519 471 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_93b9c6f870f14f6abe67530f6e6d29e8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:07.750108 471 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_93b9c6f870f14f6abe67530f6e6d29e8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:07.751056 1225 ts_tablet_manager.cc:1905] T fa47e448d3f746048afccbd2f0fe3555 P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:07.752600 1222 ts_tablet_manager.cc:1918] T 57eabef786d748e0ab8ef03b29819514 P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:07.752946 1222 log.cc:1198] T 57eabef786d748e0ab8ef03b29819514 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/57eabef786d748e0ab8ef03b29819514
I20250114 02:54:07.753544 1222 ts_tablet_manager.cc:1939] T 57eabef786d748e0ab8ef03b29819514 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:07.772569 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1cf92e5275b454ca893b12f2d90d25b" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:07.760095 1205 tablet_bootstrap.cc:492] T 57eabef786d748e0ab8ef03b29819514 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:07.774205 818 raft_consensus.cc:2463] T a1cf92e5275b454ca893b12f2d90d25b P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:07.774474 1205 ts_tablet_manager.cc:1397] T 57eabef786d748e0ab8ef03b29819514 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.044s user 0.009s sys 0.003s
I20250114 02:54:07.775534 898 leader_election.cc:304] T a1cf92e5275b454ca893b12f2d90d25b P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
I20250114 02:54:07.776487 1211 raft_consensus.cc:2798] T a1cf92e5275b454ca893b12f2d90d25b P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:07.776820 1211 raft_consensus.cc:491] T a1cf92e5275b454ca893b12f2d90d25b P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:07.777158 1211 raft_consensus.cc:3054] T a1cf92e5275b454ca893b12f2d90d25b P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:07.778138 1205 raft_consensus.cc:357] T 57eabef786d748e0ab8ef03b29819514 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:07.779126 1205 raft_consensus.cc:383] T 57eabef786d748e0ab8ef03b29819514 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:07.779516 1205 raft_consensus.cc:738] T 57eabef786d748e0ab8ef03b29819514 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:07.760053 469 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_9f231aac294f43c6baa2d1f40524bc13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:07.764925 1207 raft_consensus.cc:357] T fa47e448d3f746048afccbd2f0fe3555 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:07.780442 1207 raft_consensus.cc:383] T fa47e448d3f746048afccbd2f0fe3555 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:07.780762 1207 raft_consensus.cc:738] T fa47e448d3f746048afccbd2f0fe3555 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
W20250114 02:54:07.781440 469 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9f231aac294f43c6baa2d1f40524bc13 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:07.781772 1207 consensus_queue.cc:260] T fa47e448d3f746048afccbd2f0fe3555 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:07.773041 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1cf92e5275b454ca893b12f2d90d25b" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:07.771617 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57eabef786d748e0ab8ef03b29819514" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:07.784948 467 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet a1cf92e5275b454ca893b12f2d90d25b (table default.Table_64a15911f1d14ca882ff3b30ef9b3df9 [id=9ec503e449fb4a2a96bbb77237c2fb4f]). Expected version 2 got 0
I20250114 02:54:07.787354 1211 raft_consensus.cc:513] T a1cf92e5275b454ca893b12f2d90d25b P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:07.789604 1205 consensus_queue.cc:260] T 57eabef786d748e0ab8ef03b29819514 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:07.790975 962 raft_consensus.cc:2463] T 57eabef786d748e0ab8ef03b29819514 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 0.
I20250114 02:54:07.792250 1211 leader_election.cc:290] T a1cf92e5275b454ca893b12f2d90d25b P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Requested vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:07.793373 941 tablet_service.cc:1467] Processing CreateTablet for tablet 2837990edc4d4bea871a8114533c9352 (DEFAULT_TABLE table=default.Table_e7d101746de1409ca560602b343f7eef [id=0b1b2c3855f54d5c97bd2866dfbd8a5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:07.794410 941 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2837990edc4d4bea871a8114533c9352. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:07.796707 1205 ts_tablet_manager.cc:1428] T 57eabef786d748e0ab8ef03b29819514 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.022s user 0.008s sys 0.000s
I20250114 02:54:07.798578 1207 ts_tablet_manager.cc:1428] T fa47e448d3f746048afccbd2f0fe3555 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.072s user 0.029s sys 0.003s
I20250114 02:54:07.798683 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1cf92e5275b454ca893b12f2d90d25b" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:07.800015 751 leader_election.cc:304] T 57eabef786d748e0ab8ef03b29819514 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters: 90d6c2c58bb54e0eafb365625ac33871
W20250114 02:54:07.800696 751 raft_consensus.cc:2681] T 57eabef786d748e0ab8ef03b29819514 P 819f9e3412e54d44abb11d8b3ca877c2: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250114 02:54:07.800662 1225 ts_tablet_manager.cc:1918] T fa47e448d3f746048afccbd2f0fe3555 P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:07.801213 1225 log.cc:1198] T fa47e448d3f746048afccbd2f0fe3555 P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/fa47e448d3f746048afccbd2f0fe3555
I20250114 02:54:07.801347 1086 tablet_service.cc:1467] Processing CreateTablet for tablet 2837990edc4d4bea871a8114533c9352 (DEFAULT_TABLE table=default.Table_e7d101746de1409ca560602b343f7eef [id=0b1b2c3855f54d5c97bd2866dfbd8a5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:07.802333 1086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2837990edc4d4bea871a8114533c9352. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:07.802452 1225 ts_tablet_manager.cc:1939] T fa47e448d3f746048afccbd2f0fe3555 P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
W20250114 02:54:07.802448 898 leader_election.cc:343] T a1cf92e5275b454ca893b12f2d90d25b P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
W20250114 02:54:07.805040 898 leader_election.cc:343] T a1cf92e5275b454ca893b12f2d90d25b P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:07.804953 1085 tablet_service.cc:1467] Processing CreateTablet for tablet 6ba6181d7ee44a29a992a8d42f2b27f5 (DEFAULT_TABLE table=default.Table_f4654530e4b34b768a0b9691d76a0532 [id=e707381ee3f24b6bb3275bab2cf8fc38]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:07.805914 1085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ba6181d7ee44a29a992a8d42f2b27f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:07.803947 1179 sys_catalog.cc:455] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b303b7df086340a0b5ad6373940ae750" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b303b7df086340a0b5ad6373940ae750" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 45367 } } peers { permanent_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } } peers { permanent_uuid: "87b3db09ac87436e9a60d6de5fedaff5" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } } }
I20250114 02:54:07.806188 1179 sys_catalog.cc:458] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750 [sys.catalog]: This master's current role is: LEADER
I20250114 02:54:07.807075 1225 tablet_replica.cc:331] T 57eabef786d748e0ab8ef03b29819514 P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:07.808164 1225 raft_consensus.cc:2238] T 57eabef786d748e0ab8ef03b29819514 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:07.808594 1225 raft_consensus.cc:2267] T 57eabef786d748e0ab8ef03b29819514 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:07.809206 798 tablet_service.cc:1467] Processing CreateTablet for tablet 2837990edc4d4bea871a8114533c9352 (DEFAULT_TABLE table=default.Table_e7d101746de1409ca560602b343f7eef [id=0b1b2c3855f54d5c97bd2866dfbd8a5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:07.810307 798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2837990edc4d4bea871a8114533c9352. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:07.810626 1225 ts_tablet_manager.cc:1905] T 57eabef786d748e0ab8ef03b29819514 P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:07.813169 797 tablet_service.cc:1467] Processing CreateTablet for tablet 6ba6181d7ee44a29a992a8d42f2b27f5 (DEFAULT_TABLE table=default.Table_f4654530e4b34b768a0b9691d76a0532 [id=e707381ee3f24b6bb3275bab2cf8fc38]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:07.814477 797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ba6181d7ee44a29a992a8d42f2b27f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:07.815222 1207 tablet_bootstrap.cc:492] T a1cf92e5275b454ca893b12f2d90d25b P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:07.819523 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet fa47e448d3f746048afccbd2f0fe3555 (table default.Table_3fc4e989c384455795b08a5f0559551f [id=d2910ff0aac44e5e8f0d1ee6918255e9]) successfully deleted
I20250114 02:54:07.820911 1207 tablet_bootstrap.cc:654] T a1cf92e5275b454ca893b12f2d90d25b P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:07.792037 942 tablet_service.cc:1467] Processing CreateTablet for tablet 6ba6181d7ee44a29a992a8d42f2b27f5 (DEFAULT_TABLE table=default.Table_f4654530e4b34b768a0b9691d76a0532 [id=e707381ee3f24b6bb3275bab2cf8fc38]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:07.824177 1225 ts_tablet_manager.cc:1918] T 57eabef786d748e0ab8ef03b29819514 P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:07.824689 1225 log.cc:1198] T 57eabef786d748e0ab8ef03b29819514 P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/57eabef786d748e0ab8ef03b29819514
I20250114 02:54:07.824790 942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ba6181d7ee44a29a992a8d42f2b27f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:07.760586 465 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_f4654530e4b34b768a0b9691d76a0532" } new_table_name: "default.Table_7dcbf016e9274d768c3d742c35ce24bd" modify_external_catalogs: true
I20250114 02:54:07.833781 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet 57eabef786d748e0ab8ef03b29819514 (table default.Table_3c0778147f834283bfa9492ba941b12e [id=5b896aa662344869896925561689b5ed]) successfully deleted
I20250114 02:54:07.836611 1207 tablet_bootstrap.cc:492] T a1cf92e5275b454ca893b12f2d90d25b P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:07.836972 1225 ts_tablet_manager.cc:1939] T 57eabef786d748e0ab8ef03b29819514 P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:07.837023 1207 ts_tablet_manager.cc:1397] T a1cf92e5275b454ca893b12f2d90d25b P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.007s
I20250114 02:54:07.839514 1207 raft_consensus.cc:357] T a1cf92e5275b454ca893b12f2d90d25b P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:07.840013 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1cf92e5275b454ca893b12f2d90d25b" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:07.840188 1207 raft_consensus.cc:383] T a1cf92e5275b454ca893b12f2d90d25b P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:07.840480 1207 raft_consensus.cc:738] T a1cf92e5275b454ca893b12f2d90d25b P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:07.840562 818 raft_consensus.cc:3054] T a1cf92e5275b454ca893b12f2d90d25b P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:07.846227 466 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_dc1f5f3f0ee646889739cfb95dbcf2bb" } modify_external_catalogs: true
I20250114 02:54:07.846771 466 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_dc1f5f3f0ee646889739cfb95dbcf2bb" } modify_external_catalogs: true
I20250114 02:54:07.848661 1205 tablet_bootstrap.cc:492] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:07.848523 1207 consensus_queue.cc:260] T a1cf92e5275b454ca893b12f2d90d25b P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:07.854507 468 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_e7d101746de1409ca560602b343f7eef" } new_table_name: "default.Table_e1a99dc9b2874b3096a24cb12beb20da" modify_external_catalogs: true
I20250114 02:54:07.854750 1205 tablet_bootstrap.cc:654] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:07.858331 1207 ts_tablet_manager.cc:1428] T a1cf92e5275b454ca893b12f2d90d25b P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.021s user 0.005s sys 0.000s
I20250114 02:54:07.868057 1207 tablet_bootstrap.cc:492] T 57eabef786d748e0ab8ef03b29819514 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:07.873234 464 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_9172ebaa2b004d3083584cd19c9a2617" } modify_external_catalogs: true
I20250114 02:54:07.873723 1206 tablet_bootstrap.cc:492] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:07.873824 464 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_9172ebaa2b004d3083584cd19c9a2617" } modify_external_catalogs: true
I20250114 02:54:07.875428 1207 tablet_bootstrap.cc:654] T 57eabef786d748e0ab8ef03b29819514 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:07.879693 1206 tablet_bootstrap.cc:654] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:07.881091 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet 57eabef786d748e0ab8ef03b29819514 (table default.Table_3c0778147f834283bfa9492ba941b12e [id=5b896aa662344869896925561689b5ed]) successfully deleted
I20250114 02:54:07.881122 942 tablet_service.cc:1514] Processing DeleteTablet for tablet fa47e448d3f746048afccbd2f0fe3555 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:07 UTC) from {username='slave'} at 127.0.0.1:55068
W20250114 02:54:07.883288 456 catalog_manager.cc:4839] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet fa47e448d3f746048afccbd2f0fe3555 because the tablet was not found. No further retry: Not found: Tablet not found: fa47e448d3f746048afccbd2f0fe3555
I20250114 02:54:07.894345 1206 tablet_bootstrap.cc:492] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:07.895211 1206 ts_tablet_manager.cc:1397] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.006s
I20250114 02:54:07.895382 1207 tablet_bootstrap.cc:492] T 57eabef786d748e0ab8ef03b29819514 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:07.895845 1207 ts_tablet_manager.cc:1397] T 57eabef786d748e0ab8ef03b29819514 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.028s user 0.008s sys 0.004s
I20250114 02:54:07.897693 1206 raft_consensus.cc:357] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:07.899178 1207 raft_consensus.cc:357] T 57eabef786d748e0ab8ef03b29819514 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:07.900106 1207 raft_consensus.cc:383] T 57eabef786d748e0ab8ef03b29819514 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:07.900462 1207 raft_consensus.cc:738] T 57eabef786d748e0ab8ef03b29819514 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:07.904302 463 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet a1cf92e5275b454ca893b12f2d90d25b (table default.Table_64a15911f1d14ca882ff3b30ef9b3df9 [id=9ec503e449fb4a2a96bbb77237c2fb4f]). Expected version 2 got 0
I20250114 02:54:07.898434 1206 raft_consensus.cc:383] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:07.907116 1206 raft_consensus.cc:738] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:07.908542 1206 consensus_queue.cc:260] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:07.909013 1084 tablet_service.cc:1514] Processing DeleteTablet for tablet fa47e448d3f746048afccbd2f0fe3555 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:07 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:07.910006 1223 tablet_replica.cc:331] T fa47e448d3f746048afccbd2f0fe3555 P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:07.909933 1207 consensus_queue.cc:260] T 57eabef786d748e0ab8ef03b29819514 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:07.910977 1223 raft_consensus.cc:2238] T fa47e448d3f746048afccbd2f0fe3555 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:07.911525 1223 raft_consensus.cc:2267] T fa47e448d3f746048afccbd2f0fe3555 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:07.914278 1206 ts_tablet_manager.cc:1428] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.019s user 0.002s sys 0.004s
I20250114 02:54:07.921530 1223 ts_tablet_manager.cc:1905] T fa47e448d3f746048afccbd2f0fe3555 P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:07.924441 1207 ts_tablet_manager.cc:1428] T 57eabef786d748e0ab8ef03b29819514 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.028s user 0.009s sys 0.002s
I20250114 02:54:07.936352 942 tablet_service.cc:1467] Processing CreateTablet for tablet 96698ba107b0446cb4e847ad7a90783c (DEFAULT_TABLE table=default.Table_dc1f5f3f0ee646889739cfb95dbcf2bb [id=230ef096d1064d40af954ee8051cb4e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:07.936444 1084 tablet_service.cc:1467] Processing CreateTablet for tablet 96698ba107b0446cb4e847ad7a90783c (DEFAULT_TABLE table=default.Table_dc1f5f3f0ee646889739cfb95dbcf2bb [id=230ef096d1064d40af954ee8051cb4e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:07.937383 942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96698ba107b0446cb4e847ad7a90783c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:07.937924 1084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96698ba107b0446cb4e847ad7a90783c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:07.942448 797 tablet_service.cc:1467] Processing CreateTablet for tablet 96698ba107b0446cb4e847ad7a90783c (DEFAULT_TABLE table=default.Table_dc1f5f3f0ee646889739cfb95dbcf2bb [id=230ef096d1064d40af954ee8051cb4e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:07.943449 797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96698ba107b0446cb4e847ad7a90783c. 1 dirs total, 0 dirs full, 0 dirs failed
W20250114 02:54:07.957676 1158 tablet.cc:2367] T 57eabef786d748e0ab8ef03b29819514 P 90d6c2c58bb54e0eafb365625ac33871: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250114 02:54:07.964710 1206 tablet_bootstrap.cc:492] T 96698ba107b0446cb4e847ad7a90783c P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:07.964933 1083 tablet_service.cc:1514] Processing DeleteTablet for tablet 57eabef786d748e0ab8ef03b29819514 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:07 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:07.971222 1206 tablet_bootstrap.cc:654] T 96698ba107b0446cb4e847ad7a90783c P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:07.979844 462 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_a027831c951c4cf88c96d846f8cd360b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:07.982342 462 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a027831c951c4cf88c96d846f8cd360b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:07.947881 818 raft_consensus.cc:2463] T a1cf92e5275b454ca893b12f2d90d25b P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:08.005297 898 leader_election.cc:304] T a1cf92e5275b454ca893b12f2d90d25b P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters: 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:08.006246 1206 tablet_bootstrap.cc:492] T 96698ba107b0446cb4e847ad7a90783c P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:08.006208 1211 raft_consensus.cc:2798] T a1cf92e5275b454ca893b12f2d90d25b P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:08.006690 1206 ts_tablet_manager.cc:1397] T 96698ba107b0446cb4e847ad7a90783c P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.042s user 0.004s sys 0.009s
I20250114 02:54:08.016144 1207 tablet_bootstrap.cc:492] T 2837990edc4d4bea871a8114533c9352 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:08.017943 1206 raft_consensus.cc:357] T 96698ba107b0446cb4e847ad7a90783c P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:08.018664 1206 raft_consensus.cc:383] T 96698ba107b0446cb4e847ad7a90783c P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:08.018940 1206 raft_consensus.cc:738] T 96698ba107b0446cb4e847ad7a90783c P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:08.025977 1211 raft_consensus.cc:695] T a1cf92e5275b454ca893b12f2d90d25b P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Becoming Leader. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Running, Role: LEADER
I20250114 02:54:08.027279 1211 consensus_queue.cc:237] T a1cf92e5275b454ca893b12f2d90d25b P 525198593c8e47afaee54e74267b87e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
W20250114 02:54:08.030447 455 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.124:39105, user_credentials={real_user=slave}} blocked reactor thread for 193770us
I20250114 02:54:08.019608 1206 consensus_queue.cc:260] T 96698ba107b0446cb4e847ad7a90783c P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:08.040316 1205 tablet_bootstrap.cc:492] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:08.040930 1205 ts_tablet_manager.cc:1397] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.192s user 0.036s sys 0.091s
I20250114 02:54:08.062090 1205 raft_consensus.cc:357] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:08.063124 1205 raft_consensus.cc:383] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:08.063467 1205 raft_consensus.cc:738] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:08.065699 1206 ts_tablet_manager.cc:1428] T 96698ba107b0446cb4e847ad7a90783c P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.059s user 0.008s sys 0.000s
I20250114 02:54:08.069927 1207 tablet_bootstrap.cc:654] T 2837990edc4d4bea871a8114533c9352 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:08.075239 1205 consensus_queue.cc:260] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:08.080575 1205 ts_tablet_manager.cc:1428] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.039s user 0.004s sys 0.005s
I20250114 02:54:08.081310 471 catalog_manager.cc:5526] T a1cf92e5275b454ca893b12f2d90d25b P 525198593c8e47afaee54e74267b87e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 525198593c8e47afaee54e74267b87e3 (127.0.107.66). New cstate: current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:08.083622 471 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet a1cf92e5275b454ca893b12f2d90d25b (table default.Table_64a15911f1d14ca882ff3b30ef9b3df9 [id=9ec503e449fb4a2a96bbb77237c2fb4f]). Expected version 2 got 0
I20250114 02:54:08.091349 1205 tablet_bootstrap.cc:492] T 96698ba107b0446cb4e847ad7a90783c P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:08.094099 1223 ts_tablet_manager.cc:1918] T fa47e448d3f746048afccbd2f0fe3555 P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:08.094802 1223 log.cc:1198] T fa47e448d3f746048afccbd2f0fe3555 P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/fa47e448d3f746048afccbd2f0fe3555
I20250114 02:54:08.097381 1205 tablet_bootstrap.cc:654] T 96698ba107b0446cb4e847ad7a90783c P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:08.097908 1206 tablet_bootstrap.cc:492] T 2837990edc4d4bea871a8114533c9352 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:08.098948 1223 ts_tablet_manager.cc:1939] T fa47e448d3f746048afccbd2f0fe3555 P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
I20250114 02:54:08.102535 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet fa47e448d3f746048afccbd2f0fe3555 (table default.Table_3fc4e989c384455795b08a5f0559551f [id=d2910ff0aac44e5e8f0d1ee6918255e9]) successfully deleted
I20250114 02:54:08.104481 1207 tablet_bootstrap.cc:492] T 2837990edc4d4bea871a8114533c9352 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:08.104874 1207 ts_tablet_manager.cc:1397] T 2837990edc4d4bea871a8114533c9352 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.089s user 0.012s sys 0.001s
I20250114 02:54:08.105034 1223 tablet_replica.cc:331] T 57eabef786d748e0ab8ef03b29819514 P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:08.105855 1223 raft_consensus.cc:2238] T 57eabef786d748e0ab8ef03b29819514 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:08.114979 1223 raft_consensus.cc:2267] T 57eabef786d748e0ab8ef03b29819514 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:08.107393 1207 raft_consensus.cc:357] T 2837990edc4d4bea871a8114533c9352 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:08.115649 1207 raft_consensus.cc:383] T 2837990edc4d4bea871a8114533c9352 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:08.115953 1207 raft_consensus.cc:738] T 2837990edc4d4bea871a8114533c9352 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:08.116614 1207 consensus_queue.cc:260] T 2837990edc4d4bea871a8114533c9352 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:08.122076 1227 raft_consensus.cc:491] T 2837990edc4d4bea871a8114533c9352 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:08.122587 1227 raft_consensus.cc:513] T 2837990edc4d4bea871a8114533c9352 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:08.123283 1207 ts_tablet_manager.cc:1428] T 2837990edc4d4bea871a8114533c9352 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.018s user 0.004s sys 0.001s
I20250114 02:54:08.132561 1207 tablet_bootstrap.cc:492] T 96698ba107b0446cb4e847ad7a90783c P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:08.125133 1223 ts_tablet_manager.cc:1905] T 57eabef786d748e0ab8ef03b29819514 P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:08.141258 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2837990edc4d4bea871a8114533c9352" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:08.149333 1206 tablet_bootstrap.cc:654] T 2837990edc4d4bea871a8114533c9352 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
W20250114 02:54:08.151211 1042 leader_election.cc:343] T 2837990edc4d4bea871a8114533c9352 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:08.154301 1227 leader_election.cc:290] T 2837990edc4d4bea871a8114533c9352 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:08.162180 1205 tablet_bootstrap.cc:492] T 96698ba107b0446cb4e847ad7a90783c P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:08.162623 1205 ts_tablet_manager.cc:1397] T 96698ba107b0446cb4e847ad7a90783c P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.071s user 0.010s sys 0.016s
I20250114 02:54:08.165082 1205 raft_consensus.cc:357] T 96698ba107b0446cb4e847ad7a90783c P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:08.165776 1205 raft_consensus.cc:383] T 96698ba107b0446cb4e847ad7a90783c P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:08.166178 1205 raft_consensus.cc:738] T 96698ba107b0446cb4e847ad7a90783c P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:08.167464 1207 tablet_bootstrap.cc:654] T 96698ba107b0446cb4e847ad7a90783c P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:08.166986 1205 consensus_queue.cc:260] T 96698ba107b0446cb4e847ad7a90783c P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:08.174031 1205 ts_tablet_manager.cc:1428] T 96698ba107b0446cb4e847ad7a90783c P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.011s user 0.007s sys 0.002s
I20250114 02:54:08.179188 1205 tablet_bootstrap.cc:492] T 2837990edc4d4bea871a8114533c9352 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:08.184669 1205 tablet_bootstrap.cc:654] T 2837990edc4d4bea871a8114533c9352 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:08.189141 464 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet b2d7eaa82c6440d895ef0819e58a20ee
I20250114 02:54:08.198266 1223 ts_tablet_manager.cc:1918] T 57eabef786d748e0ab8ef03b29819514 P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:08.198632 1223 log.cc:1198] T 57eabef786d748e0ab8ef03b29819514 P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/57eabef786d748e0ab8ef03b29819514
I20250114 02:54:08.199226 1223 ts_tablet_manager.cc:1939] T 57eabef786d748e0ab8ef03b29819514 P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
I20250114 02:54:08.201848 1207 tablet_bootstrap.cc:492] T 96698ba107b0446cb4e847ad7a90783c P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:08.202232 1207 ts_tablet_manager.cc:1397] T 96698ba107b0446cb4e847ad7a90783c P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.070s user 0.005s sys 0.005s
I20250114 02:54:08.204411 1207 raft_consensus.cc:357] T 96698ba107b0446cb4e847ad7a90783c P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:08.205127 1207 raft_consensus.cc:383] T 96698ba107b0446cb4e847ad7a90783c P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:08.205407 1207 raft_consensus.cc:738] T 96698ba107b0446cb4e847ad7a90783c P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:08.206729 1205 tablet_bootstrap.cc:492] T 2837990edc4d4bea871a8114533c9352 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:08.207126 1205 ts_tablet_manager.cc:1397] T 2837990edc4d4bea871a8114533c9352 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.004s
I20250114 02:54:08.209491 1205 raft_consensus.cc:357] T 2837990edc4d4bea871a8114533c9352 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:08.210168 1205 raft_consensus.cc:383] T 2837990edc4d4bea871a8114533c9352 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:08.210429 1205 raft_consensus.cc:738] T 2837990edc4d4bea871a8114533c9352 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:08.210516 1207 consensus_queue.cc:260] T 96698ba107b0446cb4e847ad7a90783c P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:08.211166 1205 consensus_queue.cc:260] T 2837990edc4d4bea871a8114533c9352 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:08.214535 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2837990edc4d4bea871a8114533c9352" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:08.215087 962 raft_consensus.cc:2463] T 2837990edc4d4bea871a8114533c9352 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 0.
I20250114 02:54:08.216426 1040 leader_election.cc:304] T 2837990edc4d4bea871a8114533c9352 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters: 819f9e3412e54d44abb11d8b3ca877c2
I20250114 02:54:08.217540 1227 raft_consensus.cc:2798] T 2837990edc4d4bea871a8114533c9352 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:08.217833 1227 raft_consensus.cc:491] T 2837990edc4d4bea871a8114533c9352 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:08.218119 1227 raft_consensus.cc:3054] T 2837990edc4d4bea871a8114533c9352 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:08.218444 1207 ts_tablet_manager.cc:1428] T 96698ba107b0446cb4e847ad7a90783c P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.016s user 0.006s sys 0.001s
I20250114 02:54:08.222188 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet 57eabef786d748e0ab8ef03b29819514 (table default.Table_3c0778147f834283bfa9492ba941b12e [id=5b896aa662344869896925561689b5ed]) successfully deleted
I20250114 02:54:08.225247 1207 tablet_bootstrap.cc:492] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:08.225036 1227 raft_consensus.cc:513] T 2837990edc4d4bea871a8114533c9352 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:08.228883 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2837990edc4d4bea871a8114533c9352" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
I20250114 02:54:08.229635 962 raft_consensus.cc:3054] T 2837990edc4d4bea871a8114533c9352 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:08.231711 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2837990edc4d4bea871a8114533c9352" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
W20250114 02:54:08.233251 468 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2837990edc4d4bea871a8114533c9352 consensus state has no leader
I20250114 02:54:08.234103 466 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet 96698ba107b0446cb4e847ad7a90783c
I20250114 02:54:08.235913 798 tablet_service.cc:1514] Processing DeleteTablet for tablet 96698ba107b0446cb4e847ad7a90783c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:07 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:08.236733 1222 tablet_replica.cc:331] T 96698ba107b0446cb4e847ad7a90783c P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:08.237461 1222 raft_consensus.cc:2238] T 96698ba107b0446cb4e847ad7a90783c P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:08.237903 1222 raft_consensus.cc:2267] T 96698ba107b0446cb4e847ad7a90783c P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250114 02:54:08.238891 465 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6ba6181d7ee44a29a992a8d42f2b27f5 consensus state has no leader
W20250114 02:54:08.240906 1042 leader_election.cc:343] T 2837990edc4d4bea871a8114533c9352 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:08.227622 1227 leader_election.cc:290] T 2837990edc4d4bea871a8114533c9352 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Requested vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:08.241879 1205 ts_tablet_manager.cc:1428] T 2837990edc4d4bea871a8114533c9352 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.034s user 0.006s sys 0.001s
I20250114 02:54:08.242177 941 tablet_service.cc:1514] Processing DeleteTablet for tablet 96698ba107b0446cb4e847ad7a90783c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:07 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:08.243052 1225 tablet_replica.cc:331] T 96698ba107b0446cb4e847ad7a90783c P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:08.243762 1225 raft_consensus.cc:2238] T 96698ba107b0446cb4e847ad7a90783c P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:08.244203 1225 raft_consensus.cc:2267] T 96698ba107b0446cb4e847ad7a90783c P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:08.249402 1207 tablet_bootstrap.cc:654] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:08.252003 1085 tablet_service.cc:1514] Processing DeleteTablet for tablet 96698ba107b0446cb4e847ad7a90783c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:07 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:08.238571 962 raft_consensus.cc:2463] T 2837990edc4d4bea871a8114533c9352 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1.
I20250114 02:54:08.252730 1223 tablet_replica.cc:331] T 96698ba107b0446cb4e847ad7a90783c P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:08.253432 1223 raft_consensus.cc:2238] T 96698ba107b0446cb4e847ad7a90783c P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:08.253878 1223 raft_consensus.cc:2267] T 96698ba107b0446cb4e847ad7a90783c P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:08.254106 465 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 6ba6181d7ee44a29a992a8d42f2b27f5 (table default.Table_7dcbf016e9274d768c3d742c35ce24bd [id=e707381ee3f24b6bb3275bab2cf8fc38]). Expected version 1 got 0
I20250114 02:54:08.256035 1040 leader_election.cc:304] T 2837990edc4d4bea871a8114533c9352 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters: 819f9e3412e54d44abb11d8b3ca877c2
I20250114 02:54:08.257084 1227 raft_consensus.cc:2798] T 2837990edc4d4bea871a8114533c9352 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:08.265563 1222 ts_tablet_manager.cc:1905] T 96698ba107b0446cb4e847ad7a90783c P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:08.256246 1223 ts_tablet_manager.cc:1905] T 96698ba107b0446cb4e847ad7a90783c P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:08.264000 1225 ts_tablet_manager.cc:1905] T 96698ba107b0446cb4e847ad7a90783c P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:08.276377 1207 tablet_bootstrap.cc:492] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:08.276772 1207 ts_tablet_manager.cc:1397] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.052s user 0.008s sys 0.005s
I20250114 02:54:08.279450 1207 raft_consensus.cc:357] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:08.279699 465 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_9459de41277e4d2ead9f868a5aacde0e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:08.281549 465 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9459de41277e4d2ead9f868a5aacde0e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:08.284755 470 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_bf49e6df9e43470da41cf3037638033a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:08.286235 1227 raft_consensus.cc:695] T 2837990edc4d4bea871a8114533c9352 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Becoming Leader. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Running, Role: LEADER
W20250114 02:54:08.286522 470 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bf49e6df9e43470da41cf3037638033a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:08.287088 1227 consensus_queue.cc:237] T 2837990edc4d4bea871a8114533c9352 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
W20250114 02:54:08.292587 456 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 6ba6181d7ee44a29a992a8d42f2b27f5 (table default.Table_7dcbf016e9274d768c3d742c35ce24bd [id=e707381ee3f24b6bb3275bab2cf8fc38], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 6ba6181d7ee44a29a992a8d42f2b27f5 consensus state has no leader
I20250114 02:54:08.295749 1211 raft_consensus.cc:491] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:08.296480 1211 raft_consensus.cc:513] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:08.299082 1211 leader_election.cc:290] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:08.299974 1206 tablet_bootstrap.cc:492] T 2837990edc4d4bea871a8114533c9352 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:08.301399 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ba6181d7ee44a29a992a8d42f2b27f5" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:08.301971 818 raft_consensus.cc:2463] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:08.305168 898 leader_election.cc:304] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
I20250114 02:54:08.306396 1211 raft_consensus.cc:2798] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:08.306716 1211 raft_consensus.cc:491] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:08.307014 1211 raft_consensus.cc:3054] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:08.313194 1211 raft_consensus.cc:513] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:08.317034 1206 ts_tablet_manager.cc:1397] T 2837990edc4d4bea871a8114533c9352 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.219s user 0.007s sys 0.004s
I20250114 02:54:08.280087 1207 raft_consensus.cc:383] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:08.320513 1207 raft_consensus.cc:738] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:08.321131 1207 consensus_queue.cc:260] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:08.322501 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ba6181d7ee44a29a992a8d42f2b27f5" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:08.323115 1106 raft_consensus.cc:2463] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:08.323765 1207 ts_tablet_manager.cc:1428] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.047s user 0.002s sys 0.004s
I20250114 02:54:08.335287 1206 raft_consensus.cc:357] T 2837990edc4d4bea871a8114533c9352 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:08.336104 1206 raft_consensus.cc:383] T 2837990edc4d4bea871a8114533c9352 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:08.336474 1206 raft_consensus.cc:738] T 2837990edc4d4bea871a8114533c9352 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:08.338907 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ba6181d7ee44a29a992a8d42f2b27f5" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:08.339392 1106 raft_consensus.cc:3054] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:08.346089 1106 raft_consensus.cc:2463] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:08.347013 898 leader_election.cc:304] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:08.347920 1235 raft_consensus.cc:2798] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:08.340098 1211 leader_election.cc:290] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Requested vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:08.355288 470 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_fd43493446ca49a48e8a970bb92ea239"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:08.339622 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ba6181d7ee44a29a992a8d42f2b27f5" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:08.359180 818 raft_consensus.cc:3054] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:08.365233 818 raft_consensus.cc:2463] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:08.366631 1206 consensus_queue.cc:260] T 2837990edc4d4bea871a8114533c9352 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
W20250114 02:54:08.368547 470 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fd43493446ca49a48e8a970bb92ea239 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:08.370767 1212 raft_consensus.cc:491] T 2837990edc4d4bea871a8114533c9352 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:08.371160 1212 raft_consensus.cc:513] T 2837990edc4d4bea871a8114533c9352 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:08.371358 1235 raft_consensus.cc:695] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Becoming Leader. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Running, Role: LEADER
I20250114 02:54:08.372920 1212 leader_election.cc:290] T 2837990edc4d4bea871a8114533c9352 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:08.374930 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2837990edc4d4bea871a8114533c9352" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:08.376822 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2837990edc4d4bea871a8114533c9352" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:08.378404 962 raft_consensus.cc:2388] T 2837990edc4d4bea871a8114533c9352 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 819f9e3412e54d44abb11d8b3ca877c2 in current term 1: Already voted for candidate 90d6c2c58bb54e0eafb365625ac33871 in this term.
I20250114 02:54:08.379989 751 leader_election.cc:304] T 2837990edc4d4bea871a8114533c9352 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:08.381407 1206 ts_tablet_manager.cc:1428] T 2837990edc4d4bea871a8114533c9352 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.048s user 0.003s sys 0.002s
I20250114 02:54:08.381600 1212 raft_consensus.cc:3054] T 2837990edc4d4bea871a8114533c9352 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:08.378415 1223 ts_tablet_manager.cc:1918] T 96698ba107b0446cb4e847ad7a90783c P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:08.392441 1223 log.cc:1198] T 96698ba107b0446cb4e847ad7a90783c P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/96698ba107b0446cb4e847ad7a90783c
I20250114 02:54:08.395787 1222 ts_tablet_manager.cc:1918] T 96698ba107b0446cb4e847ad7a90783c P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:08.396204 1222 log.cc:1198] T 96698ba107b0446cb4e847ad7a90783c P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/96698ba107b0446cb4e847ad7a90783c
I20250114 02:54:08.397002 1222 ts_tablet_manager.cc:1939] T 96698ba107b0446cb4e847ad7a90783c P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:08.399364 1223 ts_tablet_manager.cc:1939] T 96698ba107b0446cb4e847ad7a90783c P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
I20250114 02:54:08.402590 468 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_a027831c951c4cf88c96d846f8cd360b" } modify_external_catalogs: true
I20250114 02:54:08.403188 468 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_a027831c951c4cf88c96d846f8cd360b" } modify_external_catalogs: true
I20250114 02:54:08.403278 1235 consensus_queue.cc:237] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:08.420362 1212 raft_consensus.cc:2743] T 2837990edc4d4bea871a8114533c9352 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:08.422645 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet 96698ba107b0446cb4e847ad7a90783c (table default.Table_dc1f5f3f0ee646889739cfb95dbcf2bb [id=230ef096d1064d40af954ee8051cb4e2]) successfully deleted
I20250114 02:54:08.423362 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet 96698ba107b0446cb4e847ad7a90783c (table default.Table_dc1f5f3f0ee646889739cfb95dbcf2bb [id=230ef096d1064d40af954ee8051cb4e2]) successfully deleted
I20250114 02:54:08.424846 470 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_4598efdd6ff348d081ed8c3e7dc3130f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:08.425544 1225 ts_tablet_manager.cc:1918] T 96698ba107b0446cb4e847ad7a90783c P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:08.425894 1225 log.cc:1198] T 96698ba107b0446cb4e847ad7a90783c P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/96698ba107b0446cb4e847ad7a90783c
I20250114 02:54:08.426525 1225 ts_tablet_manager.cc:1939] T 96698ba107b0446cb4e847ad7a90783c P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
W20250114 02:54:08.426487 470 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4598efdd6ff348d081ed8c3e7dc3130f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:08.429471 466 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 2837990edc4d4bea871a8114533c9352 (table default.Table_e1a99dc9b2874b3096a24cb12beb20da [id=0b1b2c3855f54d5c97bd2866dfbd8a5d]). Expected version 1 got 0
I20250114 02:54:08.432399 467 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet 2837990edc4d4bea871a8114533c9352 (table default.Table_e1a99dc9b2874b3096a24cb12beb20da [id=0b1b2c3855f54d5c97bd2866dfbd8a5d]). Expected version 1 got 0
I20250114 02:54:08.446732 465 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 2837990edc4d4bea871a8114533c9352 (table default.Table_e1a99dc9b2874b3096a24cb12beb20da [id=0b1b2c3855f54d5c97bd2866dfbd8a5d]). Expected version 1 got 0
I20250114 02:54:08.449817 471 catalog_manager.cc:5526] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 525198593c8e47afaee54e74267b87e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 525198593c8e47afaee54e74267b87e3 (127.0.107.66). New cstate: current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } }
I20250114 02:54:08.451155 471 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 6ba6181d7ee44a29a992a8d42f2b27f5 (table default.Table_7dcbf016e9274d768c3d742c35ce24bd [id=e707381ee3f24b6bb3275bab2cf8fc38]). Expected version 1 got 0
I20250114 02:54:08.457172 942 tablet_service.cc:1467] Processing CreateTablet for tablet 9ded0f4f7b0244c28d56bde1c7ec93a6 (DEFAULT_TABLE table=default.Table_a027831c951c4cf88c96d846f8cd360b [id=5c11573dbe494f24927ab32dcefded96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:08.457405 941 tablet_service.cc:1467] Processing CreateTablet for tablet 8b27d06a25264f2ebc6f6cc3dd0a2c07 (DEFAULT_TABLE table=default.Table_9f231aac294f43c6baa2d1f40524bc13 [id=a6c456511d9748f7b2c1d95ae7382f5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:08.458196 942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ded0f4f7b0244c28d56bde1c7ec93a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:08.462366 941 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b27d06a25264f2ebc6f6cc3dd0a2c07. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:08.469802 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet 96698ba107b0446cb4e847ad7a90783c (table default.Table_dc1f5f3f0ee646889739cfb95dbcf2bb [id=230ef096d1064d40af954ee8051cb4e2]) successfully deleted
I20250114 02:54:08.478425 1106 raft_consensus.cc:3054] T a1cf92e5275b454ca893b12f2d90d25b P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:08.482381 940 tablet_service.cc:1467] Processing CreateTablet for tablet 54f9bb2d2e334f53b0c0560f26bbfa3f (DEFAULT_TABLE table=default.Table_c6640103d84c48beb554d7509b246b27 [id=4873b55a952644e3919e7759a9c59b07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:08.483479 940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54f9bb2d2e334f53b0c0560f26bbfa3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:08.485452 1106 raft_consensus.cc:1270] T a1cf92e5275b454ca893b12f2d90d25b P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Refusing update from remote peer 525198593c8e47afaee54e74267b87e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 02:54:08.485497 939 tablet_service.cc:1467] Processing CreateTablet for tablet 2a88342b30704cbda0f11f9ddc312fab (DEFAULT_TABLE table=default.Table_93b9c6f870f14f6abe67530f6e6d29e8 [id=b1e4e75afa524c68b5976a885116ef1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:08.486971 939 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a88342b30704cbda0f11f9ddc312fab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:08.487552 1235 consensus_queue.cc:1035] T a1cf92e5275b454ca893b12f2d90d25b P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:08.493937 1205 tablet_bootstrap.cc:492] T 9ded0f4f7b0244c28d56bde1c7ec93a6 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:08.495282 818 raft_consensus.cc:1270] T a1cf92e5275b454ca893b12f2d90d25b P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Refusing update from remote peer 525198593c8e47afaee54e74267b87e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 02:54:08.497205 1235 consensus_queue.cc:1035] T a1cf92e5275b454ca893b12f2d90d25b P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:08.498667 1085 tablet_service.cc:1514] Processing DeleteTablet for tablet 96698ba107b0446cb4e847ad7a90783c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:07 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:08.499318 1205 tablet_bootstrap.cc:654] T 9ded0f4f7b0244c28d56bde1c7ec93a6 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:08.502807 798 tablet_service.cc:1467] Processing CreateTablet for tablet 8b27d06a25264f2ebc6f6cc3dd0a2c07 (DEFAULT_TABLE table=default.Table_9f231aac294f43c6baa2d1f40524bc13 [id=a6c456511d9748f7b2c1d95ae7382f5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:08.504112 798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b27d06a25264f2ebc6f6cc3dd0a2c07. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:08.529954 1205 tablet_bootstrap.cc:492] T 9ded0f4f7b0244c28d56bde1c7ec93a6 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:08.530422 1205 ts_tablet_manager.cc:1397] T 9ded0f4f7b0244c28d56bde1c7ec93a6 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.037s user 0.009s sys 0.003s
I20250114 02:54:08.554004 1258 tablet.cc:1721] T a1cf92e5275b454ca893b12f2d90d25b P 819f9e3412e54d44abb11d8b3ca877c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250114 02:54:08.557545 1085 tablet_service.cc:1467] Processing CreateTablet for tablet 8b27d06a25264f2ebc6f6cc3dd0a2c07 (DEFAULT_TABLE table=default.Table_9f231aac294f43c6baa2d1f40524bc13 [id=a6c456511d9748f7b2c1d95ae7382f5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:08.557798 797 tablet_service.cc:1467] Processing CreateTablet for tablet 9ded0f4f7b0244c28d56bde1c7ec93a6 (DEFAULT_TABLE table=default.Table_a027831c951c4cf88c96d846f8cd360b [id=5c11573dbe494f24927ab32dcefded96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:08.558643 1085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b27d06a25264f2ebc6f6cc3dd0a2c07. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:08.558733 797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ded0f4f7b0244c28d56bde1c7ec93a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:08.565937 1084 tablet_service.cc:1467] Processing CreateTablet for tablet 9ded0f4f7b0244c28d56bde1c7ec93a6 (DEFAULT_TABLE table=default.Table_a027831c951c4cf88c96d846f8cd360b [id=5c11573dbe494f24927ab32dcefded96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:08.567291 1084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ded0f4f7b0244c28d56bde1c7ec93a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:08.567117 1257 tablet.cc:1721] T a1cf92e5275b454ca893b12f2d90d25b P 525198593c8e47afaee54e74267b87e3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250114 02:54:08.570420 1205 raft_consensus.cc:357] T 9ded0f4f7b0244c28d56bde1c7ec93a6 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:08.571175 1205 raft_consensus.cc:383] T 9ded0f4f7b0244c28d56bde1c7ec93a6 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:08.571430 1205 raft_consensus.cc:738] T 9ded0f4f7b0244c28d56bde1c7ec93a6 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:08.571991 1205 consensus_queue.cc:260] T 9ded0f4f7b0244c28d56bde1c7ec93a6 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:08.587481 1207 tablet_bootstrap.cc:492] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:08.591935 796 tablet_service.cc:1467] Processing CreateTablet for tablet 54f9bb2d2e334f53b0c0560f26bbfa3f (DEFAULT_TABLE table=default.Table_c6640103d84c48beb554d7509b246b27 [id=4873b55a952644e3919e7759a9c59b07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:08.593187 796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54f9bb2d2e334f53b0c0560f26bbfa3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:08.597783 1207 tablet_bootstrap.cc:654] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:08.598907 817 raft_consensus.cc:1270] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Refusing update from remote peer 525198593c8e47afaee54e74267b87e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 02:54:08.600495 1235 consensus_queue.cc:1035] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:08.609416 1252 mvcc.cc:204] Tried to move back new op lower bound from 7114028025732173824 to 7114028023951065088. Current Snapshot: MvccSnapshot[applied={T|T < 7114028025732173824}]
I20250114 02:54:08.609652 1105 raft_consensus.cc:1270] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Refusing update from remote peer 525198593c8e47afaee54e74267b87e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 02:54:08.611407 1205 ts_tablet_manager.cc:1428] T 9ded0f4f7b0244c28d56bde1c7ec93a6 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.081s user 0.007s sys 0.000s
I20250114 02:54:08.613078 1249 consensus_queue.cc:1035] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:54:08.615913 1086 tablet_service.cc:1467] Processing CreateTablet for tablet 2a88342b30704cbda0f11f9ddc312fab (DEFAULT_TABLE table=default.Table_93b9c6f870f14f6abe67530f6e6d29e8 [id=b1e4e75afa524c68b5976a885116ef1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:08.617015 1086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a88342b30704cbda0f11f9ddc312fab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:08.618130 795 tablet_service.cc:1467] Processing CreateTablet for tablet 2a88342b30704cbda0f11f9ddc312fab (DEFAULT_TABLE table=default.Table_93b9c6f870f14f6abe67530f6e6d29e8 [id=b1e4e75afa524c68b5976a885116ef1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:08.619244 795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a88342b30704cbda0f11f9ddc312fab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:08.619056 1085 tablet_service.cc:1467] Processing CreateTablet for tablet 54f9bb2d2e334f53b0c0560f26bbfa3f (DEFAULT_TABLE table=default.Table_c6640103d84c48beb554d7509b246b27 [id=4873b55a952644e3919e7759a9c59b07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:08.620090 1085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54f9bb2d2e334f53b0c0560f26bbfa3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:08.623306 466 catalog_manager.cc:5526] T 2837990edc4d4bea871a8114533c9352 P 90d6c2c58bb54e0eafb365625ac33871 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67). New cstate: current_term: 1 leader_uuid: "90d6c2c58bb54e0eafb365625ac33871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:08.625250 466 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet 2837990edc4d4bea871a8114533c9352 (table default.Table_e1a99dc9b2874b3096a24cb12beb20da [id=0b1b2c3855f54d5c97bd2866dfbd8a5d]). Expected version 1 got 0
I20250114 02:54:08.625937 466 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet 6ba6181d7ee44a29a992a8d42f2b27f5 (table default.Table_7dcbf016e9274d768c3d742c35ce24bd [id=e707381ee3f24b6bb3275bab2cf8fc38]). Expected version 1 got 0
I20250114 02:54:08.626451 471 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet a1cf92e5275b454ca893b12f2d90d25b (table default.Table_64a15911f1d14ca882ff3b30ef9b3df9 [id=9ec503e449fb4a2a96bbb77237c2fb4f]). Expected version 2 got 0
I20250114 02:54:08.628988 468 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet 9ded0f4f7b0244c28d56bde1c7ec93a6
I20250114 02:54:08.631853 942 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ded0f4f7b0244c28d56bde1c7ec93a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:08 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:08.632823 1225 tablet_replica.cc:331] T 9ded0f4f7b0244c28d56bde1c7ec93a6 P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:08.633512 1225 raft_consensus.cc:2238] T 9ded0f4f7b0244c28d56bde1c7ec93a6 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:08.633945 1225 raft_consensus.cc:2267] T 9ded0f4f7b0244c28d56bde1c7ec93a6 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:08.635699 470 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_e1a99dc9b2874b3096a24cb12beb20da" } new_table_name: "default.Table_1f95cb34af29429a9749250aec5a5af0" modify_external_catalogs: true
I20250114 02:54:08.637300 467 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_eb42bfe44f044d09b56f6961ccf64c06"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:08.638559 1225 ts_tablet_manager.cc:1905] T 9ded0f4f7b0244c28d56bde1c7ec93a6 P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250114 02:54:08.638922 467 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eb42bfe44f044d09b56f6961ccf64c06 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:08.670040 1251 mvcc.cc:204] Tried to move back new op lower bound from 7114028026161799168 to 7114028025533558784. Current Snapshot: MvccSnapshot[applied={T|T < 7114028026161799168}]
I20250114 02:54:08.670440 471 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_9459de41277e4d2ead9f868a5aacde0e" } modify_external_catalogs: true
I20250114 02:54:08.671025 471 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_9459de41277e4d2ead9f868a5aacde0e" } modify_external_catalogs: true
I20250114 02:54:08.672605 1253 mvcc.cc:204] Tried to move back new op lower bound from 7114028026161799168 to 7114028025533558784. Current Snapshot: MvccSnapshot[applied={T|T < 7114028026161799168}]
I20250114 02:54:08.676668 468 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_9f231aac294f43c6baa2d1f40524bc13" } new_table_name: "default.Table_480725b9282e4dd7a13623a6dab4f9ee" modify_external_catalogs: true
I20250114 02:54:08.693992 794 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ded0f4f7b0244c28d56bde1c7ec93a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:08 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:08.696364 1083 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ded0f4f7b0244c28d56bde1c7ec93a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:08 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:08.698328 464 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_d736c43978ca4363949e814cc78b057e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:08.700047 464 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d736c43978ca4363949e814cc78b057e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:08.701617 1259 tablet.cc:1721] T a1cf92e5275b454ca893b12f2d90d25b P 90d6c2c58bb54e0eafb365625ac33871: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250114 02:54:08.703554 462 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: default.Table_64a15911f1d14ca882ff3b30ef9b3df9 [id=9ec503e449fb4a2a96bbb77237c2fb4f] alter complete (version 2)
W20250114 02:54:08.705687 458 catalog_manager.cc:4839] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 96698ba107b0446cb4e847ad7a90783c because the tablet was not found. No further retry: Not found: Tablet not found: 96698ba107b0446cb4e847ad7a90783c
W20250114 02:54:08.709245 458 catalog_manager.cc:4839] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 9ded0f4f7b0244c28d56bde1c7ec93a6 because the tablet was not found. No further retry: Not found: Tablet not found: 9ded0f4f7b0244c28d56bde1c7ec93a6
W20250114 02:54:08.710155 458 catalog_manager.cc:4839] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet 9ded0f4f7b0244c28d56bde1c7ec93a6 because the tablet was not found. No further retry: Not found: Tablet not found: 9ded0f4f7b0244c28d56bde1c7ec93a6
I20250114 02:54:08.725106 1206 tablet_bootstrap.cc:492] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:08.730300 1205 tablet_bootstrap.cc:492] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:08.734356 1206 tablet_bootstrap.cc:654] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:08.736666 1205 tablet_bootstrap.cc:654] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:08.744333 1262 tablet.cc:1721] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 819f9e3412e54d44abb11d8b3ca877c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250114 02:54:08.747314 1257 tablet.cc:1721] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 525198593c8e47afaee54e74267b87e3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250114 02:54:08.758692 1207 tablet_bootstrap.cc:492] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:08.759140 1207 ts_tablet_manager.cc:1397] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.172s user 0.008s sys 0.012s
I20250114 02:54:08.776662 465 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_97368d1825cf41e981b17a5e7c8d2148"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:08.778770 465 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_97368d1825cf41e981b17a5e7c8d2148 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:08.791169 1207 raft_consensus.cc:357] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:08.791893 1207 raft_consensus.cc:383] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:08.792167 1207 raft_consensus.cc:738] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:08.792289 1225 ts_tablet_manager.cc:1918] T 9ded0f4f7b0244c28d56bde1c7ec93a6 P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:08.792851 1207 consensus_queue.cc:260] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:08.796146 1207 ts_tablet_manager.cc:1428] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.037s user 0.003s sys 0.004s
I20250114 02:54:08.797277 1207 tablet_bootstrap.cc:492] T 2a88342b30704cbda0f11f9ddc312fab P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:08.798308 1206 tablet_bootstrap.cc:492] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:08.798682 1206 ts_tablet_manager.cc:1397] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.074s user 0.010s sys 0.015s
I20250114 02:54:08.801158 1206 raft_consensus.cc:357] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:08.801753 1206 raft_consensus.cc:383] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:08.801990 1206 raft_consensus.cc:738] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:08.802294 1207 tablet_bootstrap.cc:654] T 2a88342b30704cbda0f11f9ddc312fab P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:08.802657 1206 consensus_queue.cc:260] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:08.808526 471 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet bc1ed340c98149458b5103bdd935e289
I20250114 02:54:08.811436 1206 ts_tablet_manager.cc:1428] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.012s user 0.004s sys 0.000s
I20250114 02:54:08.816848 1225 log.cc:1198] T 9ded0f4f7b0244c28d56bde1c7ec93a6 P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/9ded0f4f7b0244c28d56bde1c7ec93a6
I20250114 02:54:08.818739 1206 tablet_bootstrap.cc:492] T 9ded0f4f7b0244c28d56bde1c7ec93a6 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:08.818807 1225 ts_tablet_manager.cc:1939] T 9ded0f4f7b0244c28d56bde1c7ec93a6 P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:08.825026 1206 tablet_bootstrap.cc:654] T 9ded0f4f7b0244c28d56bde1c7ec93a6 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:08.826977 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet 9ded0f4f7b0244c28d56bde1c7ec93a6 (table default.Table_a027831c951c4cf88c96d846f8cd360b [id=5c11573dbe494f24927ab32dcefded96]) successfully deleted
I20250114 02:54:08.835305 1259 tablet.cc:1721] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 90d6c2c58bb54e0eafb365625ac33871: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250114 02:54:08.841939 468 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8b27d06a25264f2ebc6f6cc3dd0a2c07 consensus state has no leader
I20250114 02:54:08.855471 962 raft_consensus.cc:1270] T 2837990edc4d4bea871a8114533c9352 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Refusing update from remote peer 90d6c2c58bb54e0eafb365625ac33871: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 02:54:08.856114 818 raft_consensus.cc:1270] T 2837990edc4d4bea871a8114533c9352 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Refusing update from remote peer 90d6c2c58bb54e0eafb365625ac33871: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 02:54:08.857751 1227 consensus_queue.cc:1035] T 2837990edc4d4bea871a8114533c9352 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:08.859144 1227 consensus_queue.cc:1035] T 2837990edc4d4bea871a8114533c9352 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:08.870658 1205 tablet_bootstrap.cc:492] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:08.871059 1205 ts_tablet_manager.cc:1397] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.141s user 0.008s sys 0.000s
I20250114 02:54:08.873401 1205 raft_consensus.cc:357] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:08.874434 1205 raft_consensus.cc:383] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:08.874811 1205 raft_consensus.cc:738] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:08.875613 1205 consensus_queue.cc:260] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:08.878181 1205 ts_tablet_manager.cc:1428] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250114 02:54:08.897672 465 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_d12c2d9a4b3f42d4839548ca62645693"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:08.900883 465 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d12c2d9a4b3f42d4839548ca62645693 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:08.907089 464 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_480725b9282e4dd7a13623a6dab4f9ee" } modify_external_catalogs: true
I20250114 02:54:08.907447 1205 tablet_bootstrap.cc:492] T 2a88342b30704cbda0f11f9ddc312fab P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:08.907639 464 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_480725b9282e4dd7a13623a6dab4f9ee" } modify_external_catalogs: true
I20250114 02:54:08.933600 462 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 2837990edc4d4bea871a8114533c9352 (table default.Table_e1a99dc9b2874b3096a24cb12beb20da [id=0b1b2c3855f54d5c97bd2866dfbd8a5d]). Expected version 1 got 0
I20250114 02:54:08.934394 462 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 6ba6181d7ee44a29a992a8d42f2b27f5 (table default.Table_7dcbf016e9274d768c3d742c35ce24bd [id=e707381ee3f24b6bb3275bab2cf8fc38]). Expected version 1 got 0
I20250114 02:54:08.940799 467 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: default.Table_7dcbf016e9274d768c3d742c35ce24bd [id=e707381ee3f24b6bb3275bab2cf8fc38] alter complete (version 1)
I20250114 02:54:08.942489 1205 tablet_bootstrap.cc:654] T 2a88342b30704cbda0f11f9ddc312fab P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:08.947362 1206 tablet_bootstrap.cc:492] T 9ded0f4f7b0244c28d56bde1c7ec93a6 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:08.947894 1206 ts_tablet_manager.cc:1397] T 9ded0f4f7b0244c28d56bde1c7ec93a6 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.129s user 0.034s sys 0.040s
I20250114 02:54:08.950727 1206 raft_consensus.cc:357] T 9ded0f4f7b0244c28d56bde1c7ec93a6 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:08.951599 1206 raft_consensus.cc:383] T 9ded0f4f7b0244c28d56bde1c7ec93a6 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:08.951915 1206 raft_consensus.cc:738] T 9ded0f4f7b0244c28d56bde1c7ec93a6 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:08.952647 1206 consensus_queue.cc:260] T 9ded0f4f7b0244c28d56bde1c7ec93a6 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:08.956125 1206 ts_tablet_manager.cc:1428] T 9ded0f4f7b0244c28d56bde1c7ec93a6 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20250114 02:54:08.957006 1206 tablet_bootstrap.cc:492] T 2a88342b30704cbda0f11f9ddc312fab P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:08.962828 940 tablet_service.cc:1467] Processing CreateTablet for tablet a456420f78714cf895528e19dfe0ee8d (DEFAULT_TABLE table=default.Table_fd43493446ca49a48e8a970bb92ea239 [id=6dbd2db73b5443a2af5d3b5056ab0169]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:08.962807 798 tablet_service.cc:1467] Processing CreateTablet for tablet a456420f78714cf895528e19dfe0ee8d (DEFAULT_TABLE table=default.Table_fd43493446ca49a48e8a970bb92ea239 [id=6dbd2db73b5443a2af5d3b5056ab0169]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:08.963907 940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a456420f78714cf895528e19dfe0ee8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:08.965160 796 tablet_service.cc:1467] Processing CreateTablet for tablet ee4ed8205f5641bf96b3f617d7f81321 (DEFAULT_TABLE table=default.Table_bf49e6df9e43470da41cf3037638033a [id=e8c1f073ed504963947cab370eb9cf1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:08.964190 798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a456420f78714cf895528e19dfe0ee8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:08.971432 941 tablet_service.cc:1467] Processing CreateTablet for tablet ee4ed8205f5641bf96b3f617d7f81321 (DEFAULT_TABLE table=default.Table_bf49e6df9e43470da41cf3037638033a [id=e8c1f073ed504963947cab370eb9cf1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:08.973790 941 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee4ed8205f5641bf96b3f617d7f81321. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:08.983479 796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee4ed8205f5641bf96b3f617d7f81321. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:08.983665 1205 tablet_bootstrap.cc:492] T 2a88342b30704cbda0f11f9ddc312fab P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:08.984455 1205 ts_tablet_manager.cc:1397] T 2a88342b30704cbda0f11f9ddc312fab P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.077s user 0.013s sys 0.000s
I20250114 02:54:08.986908 1205 raft_consensus.cc:357] T 2a88342b30704cbda0f11f9ddc312fab P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:08.987519 1205 raft_consensus.cc:383] T 2a88342b30704cbda0f11f9ddc312fab P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:08.987769 1205 raft_consensus.cc:738] T 2a88342b30704cbda0f11f9ddc312fab P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:08.988323 1205 consensus_queue.cc:260] T 2a88342b30704cbda0f11f9ddc312fab P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:08.994205 1205 ts_tablet_manager.cc:1428] T 2a88342b30704cbda0f11f9ddc312fab P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20250114 02:54:08.995013 1205 tablet_bootstrap.cc:492] T 54f9bb2d2e334f53b0c0560f26bbfa3f P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:09.008919 470 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 2837990edc4d4bea871a8114533c9352 (table default.Table_1f95cb34af29429a9749250aec5a5af0 [id=0b1b2c3855f54d5c97bd2866dfbd8a5d]). Expected version 2 got 0
I20250114 02:54:09.013072 465 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 2837990edc4d4bea871a8114533c9352 (table default.Table_1f95cb34af29429a9749250aec5a5af0 [id=0b1b2c3855f54d5c97bd2866dfbd8a5d]). Expected version 2 got 0
I20250114 02:54:09.015269 466 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_c6640103d84c48beb554d7509b246b27" } modify_external_catalogs: true
I20250114 02:54:09.015985 1206 tablet_bootstrap.cc:654] T 2a88342b30704cbda0f11f9ddc312fab P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:09.016095 466 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_c6640103d84c48beb554d7509b246b27" } modify_external_catalogs: true
I20250114 02:54:09.016497 464 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet 8b27d06a25264f2ebc6f6cc3dd0a2c07
I20250114 02:54:09.019058 939 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b27d06a25264f2ebc6f6cc3dd0a2c07 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:08 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:09.019732 1225 tablet_replica.cc:331] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:09.020488 1225 raft_consensus.cc:2238] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:09.021445 1205 tablet_bootstrap.cc:654] T 54f9bb2d2e334f53b0c0560f26bbfa3f P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:09.021489 1225 raft_consensus.cc:2267] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:09.021831 1274 raft_consensus.cc:491] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:09.022331 1274 raft_consensus.cc:513] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:09.023851 1225 ts_tablet_manager.cc:1905] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:09.025383 1274 leader_election.cc:290] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:09.025252 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b27d06a25264f2ebc6f6cc3dd0a2c07" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:09.025853 818 raft_consensus.cc:2463] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 0.
I20250114 02:54:09.027503 1042 leader_election.cc:304] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:09.028292 1227 raft_consensus.cc:2798] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:09.030938 1258 tablet.cc:1721] T 2837990edc4d4bea871a8114533c9352 P 819f9e3412e54d44abb11d8b3ca877c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250114 02:54:09.032444 795 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b27d06a25264f2ebc6f6cc3dd0a2c07 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:08 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:09.034183 1222 tablet_replica.cc:331] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:09.034969 1222 raft_consensus.cc:2238] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:09.035404 1222 raft_consensus.cc:2267] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:09.035614 1227 raft_consensus.cc:491] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:09.035950 1227 raft_consensus.cc:3054] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:09.041177 1207 tablet_bootstrap.cc:492] T 2a88342b30704cbda0f11f9ddc312fab P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:09.041553 1207 ts_tablet_manager.cc:1397] T 2a88342b30704cbda0f11f9ddc312fab P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.245s user 0.059s sys 0.086s
I20250114 02:54:09.048188 462 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_5b40399263bc473ca44553f9344f048a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:09.048794 1222 ts_tablet_manager.cc:1905] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250114 02:54:09.050628 462 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5b40399263bc473ca44553f9344f048a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:09.052114 795 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ded0f4f7b0244c28d56bde1c7ec93a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:08 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:09.053341 470 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_24dc0a51cadc4387918d43a65986ebf4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:09.054986 470 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_24dc0a51cadc4387918d43a65986ebf4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:09.055024 468 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 2837990edc4d4bea871a8114533c9352 (table default.Table_1f95cb34af29429a9749250aec5a5af0 [id=0b1b2c3855f54d5c97bd2866dfbd8a5d]). Expected version 2 got 1
I20250114 02:54:09.057631 1259 tablet.cc:1721] T 2837990edc4d4bea871a8114533c9352 P 90d6c2c58bb54e0eafb365625ac33871: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250114 02:54:09.066797 1040 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.66:44037, user_credentials={real_user=slave}} blocked reactor thread for 83372.6us
I20250114 02:54:09.068459 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b27d06a25264f2ebc6f6cc3dd0a2c07" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:09.073433 1257 tablet.cc:1721] T 2837990edc4d4bea871a8114533c9352 P 525198593c8e47afaee54e74267b87e3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250114 02:54:09.079001 467 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_eb42bfe44f044d09b56f6961ccf64c06" } new_table_name: "default.Table_017604272128419e9c9d1bba85e595cb" modify_external_catalogs: true
I20250114 02:54:09.085592 1085 tablet_service.cc:1467] Processing CreateTablet for tablet a456420f78714cf895528e19dfe0ee8d (DEFAULT_TABLE table=default.Table_fd43493446ca49a48e8a970bb92ea239 [id=6dbd2db73b5443a2af5d3b5056ab0169]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:09.086794 1085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a456420f78714cf895528e19dfe0ee8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:09.091547 466 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet 54f9bb2d2e334f53b0c0560f26bbfa3f
I20250114 02:54:09.094494 1086 tablet_service.cc:1467] Processing CreateTablet for tablet ee4ed8205f5641bf96b3f617d7f81321 (DEFAULT_TABLE table=default.Table_bf49e6df9e43470da41cf3037638033a [id=e8c1f073ed504963947cab370eb9cf1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:09.095660 1086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee4ed8205f5641bf96b3f617d7f81321. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:09.100492 1206 tablet_bootstrap.cc:492] T 2a88342b30704cbda0f11f9ddc312fab P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:09.100513 468 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 2837990edc4d4bea871a8114533c9352 (table default.Table_1f95cb34af29429a9749250aec5a5af0 [id=0b1b2c3855f54d5c97bd2866dfbd8a5d]). Expected version 2 got 1
I20250114 02:54:09.101186 1206 ts_tablet_manager.cc:1397] T 2a88342b30704cbda0f11f9ddc312fab P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.144s user 0.014s sys 0.003s
I20250114 02:54:09.102630 465 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 2837990edc4d4bea871a8114533c9352 (table default.Table_1f95cb34af29429a9749250aec5a5af0 [id=0b1b2c3855f54d5c97bd2866dfbd8a5d]). Expected version 2 got 1
I20250114 02:54:09.109582 941 tablet_service.cc:1514] Processing DeleteTablet for tablet 54f9bb2d2e334f53b0c0560f26bbfa3f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:09 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:09.110370 796 tablet_service.cc:1514] Processing DeleteTablet for tablet 54f9bb2d2e334f53b0c0560f26bbfa3f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:09 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:09.111687 464 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_075cadc0bfa145718aa9f2f6850f595c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:09.114519 464 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_075cadc0bfa145718aa9f2f6850f595c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:09.115079 1084 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b27d06a25264f2ebc6f6cc3dd0a2c07 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:08 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:09.115814 1223 tablet_replica.cc:331] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:09.116253 1084 tablet_service.cc:1514] Processing DeleteTablet for tablet 54f9bb2d2e334f53b0c0560f26bbfa3f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:09 UTC) from {username='slave'} at 127.0.0.1:53876
W20250114 02:54:09.116981 1040 leader_election.cc:343] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: Tablet not RUNNING: STOPPED
I20250114 02:54:09.110146 1206 raft_consensus.cc:357] T 2a88342b30704cbda0f11f9ddc312fab P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:09.119544 1206 raft_consensus.cc:383] T 2a88342b30704cbda0f11f9ddc312fab P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:09.122678 1206 raft_consensus.cc:738] T 2a88342b30704cbda0f11f9ddc312fab P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
W20250114 02:54:09.125182 467 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ba074d4ea17747d5aa9287bdaf541681 has no consensus state
I20250114 02:54:09.133173 467 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet 2837990edc4d4bea871a8114533c9352 (table default.Table_1f95cb34af29429a9749250aec5a5af0 [id=0b1b2c3855f54d5c97bd2866dfbd8a5d]). Expected version 2 got 1
I20250114 02:54:09.134341 1222 ts_tablet_manager.cc:1918] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:09.129503 1206 consensus_queue.cc:260] T 2a88342b30704cbda0f11f9ddc312fab P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:09.139412 1222 log.cc:1198] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/8b27d06a25264f2ebc6f6cc3dd0a2c07
I20250114 02:54:09.147864 1225 ts_tablet_manager.cc:1918] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:09.149722 1225 log.cc:1198] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/8b27d06a25264f2ebc6f6cc3dd0a2c07
I20250114 02:54:09.150372 1225 ts_tablet_manager.cc:1939] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:09.153108 1222 ts_tablet_manager.cc:1939] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:09.043767 1207 raft_consensus.cc:357] T 2a88342b30704cbda0f11f9ddc312fab P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:09.156631 1207 raft_consensus.cc:383] T 2a88342b30704cbda0f11f9ddc312fab P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:09.156921 1207 raft_consensus.cc:738] T 2a88342b30704cbda0f11f9ddc312fab P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:09.157322 466 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_64a15911f1d14ca882ff3b30ef9b3df9" } new_table_name: "default.Table_63284ca278eb44d8b01f8e7c34a7a6e3" modify_external_catalogs: true
I20250114 02:54:09.157560 1207 consensus_queue.cc:260] T 2a88342b30704cbda0f11f9ddc312fab P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:09.159181 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet 8b27d06a25264f2ebc6f6cc3dd0a2c07 (table default.Table_480725b9282e4dd7a13623a6dab4f9ee [id=a6c456511d9748f7b2c1d95ae7382f5d]) successfully deleted
I20250114 02:54:09.159901 1207 ts_tablet_manager.cc:1428] T 2a88342b30704cbda0f11f9ddc312fab P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.118s user 0.005s sys 0.001s
I20250114 02:54:09.159423 1206 ts_tablet_manager.cc:1428] T 2a88342b30704cbda0f11f9ddc312fab P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.058s user 0.007s sys 0.002s
W20250114 02:54:09.160444 456 catalog_manager.cc:4852] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet 54f9bb2d2e334f53b0c0560f26bbfa3f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 54f9bb2d2e334f53b0c0560f26bbfa3f already in progress: creating tablet
I20250114 02:54:09.160795 1206 tablet_bootstrap.cc:492] T 54f9bb2d2e334f53b0c0560f26bbfa3f P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:09.174432 1205 tablet_bootstrap.cc:492] T 54f9bb2d2e334f53b0c0560f26bbfa3f P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:09.174834 1205 ts_tablet_manager.cc:1397] T 54f9bb2d2e334f53b0c0560f26bbfa3f P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.180s user 0.017s sys 0.005s
I20250114 02:54:09.177006 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet 8b27d06a25264f2ebc6f6cc3dd0a2c07 (table default.Table_480725b9282e4dd7a13623a6dab4f9ee [id=a6c456511d9748f7b2c1d95ae7382f5d]) successfully deleted
I20250114 02:54:09.178923 468 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 2837990edc4d4bea871a8114533c9352 (table default.Table_1f95cb34af29429a9749250aec5a5af0 [id=0b1b2c3855f54d5c97bd2866dfbd8a5d]). Expected version 2 got 1
I20250114 02:54:09.180279 1222 tablet_replica.cc:331] T 9ded0f4f7b0244c28d56bde1c7ec93a6 P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:09.183578 1222 raft_consensus.cc:2238] T 9ded0f4f7b0244c28d56bde1c7ec93a6 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:09.184119 1222 raft_consensus.cc:2267] T 9ded0f4f7b0244c28d56bde1c7ec93a6 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:09.164094 1227 raft_consensus.cc:513] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:09.187072 1223 raft_consensus.cc:2238] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:09.188309 1205 raft_consensus.cc:357] T 54f9bb2d2e334f53b0c0560f26bbfa3f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:09.188957 1205 raft_consensus.cc:383] T 54f9bb2d2e334f53b0c0560f26bbfa3f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:09.189200 1205 raft_consensus.cc:738] T 54f9bb2d2e334f53b0c0560f26bbfa3f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:09.189661 1223 raft_consensus.cc:2267] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:09.189735 1205 consensus_queue.cc:260] T 54f9bb2d2e334f53b0c0560f26bbfa3f P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:09.202294 464 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_55dad15f1b4f4e22854ba3070b0eea8e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:09.204035 464 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_55dad15f1b4f4e22854ba3070b0eea8e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:09.191582 1205 ts_tablet_manager.cc:1428] T 54f9bb2d2e334f53b0c0560f26bbfa3f P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.016s user 0.004s sys 0.000s
I20250114 02:54:09.203907 1222 ts_tablet_manager.cc:1905] T 9ded0f4f7b0244c28d56bde1c7ec93a6 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250114 02:54:09.212049 1040 leader_election.cc:343] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Not found: Tablet not found: 8b27d06a25264f2ebc6f6cc3dd0a2c07
W20250114 02:54:09.212495 1042 leader_election.cc:343] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Not found: Tablet not found: 8b27d06a25264f2ebc6f6cc3dd0a2c07
I20250114 02:54:09.212870 1042 leader_election.cc:304] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90d6c2c58bb54e0eafb365625ac33871; no voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2
I20250114 02:54:09.209522 1206 tablet_bootstrap.cc:654] T 54f9bb2d2e334f53b0c0560f26bbfa3f P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
W20250114 02:54:09.213505 1042 raft_consensus.cc:2681] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 90d6c2c58bb54e0eafb365625ac33871: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250114 02:54:09.214327 1207 tablet_bootstrap.cc:492] T 9ded0f4f7b0244c28d56bde1c7ec93a6 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:09.213997 1205 tablet_bootstrap.cc:492] T ee4ed8205f5641bf96b3f617d7f81321 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:09.215878 1249 raft_consensus.cc:491] T 2a88342b30704cbda0f11f9ddc312fab P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:09.216346 1249 raft_consensus.cc:513] T 2a88342b30704cbda0f11f9ddc312fab P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:09.218786 1249 leader_election.cc:290] T 2a88342b30704cbda0f11f9ddc312fab P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:09.220700 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a88342b30704cbda0f11f9ddc312fab" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:09.221871 818 raft_consensus.cc:2463] T 2a88342b30704cbda0f11f9ddc312fab P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:09.223409 467 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet 2837990edc4d4bea871a8114533c9352 (table default.Table_1f95cb34af29429a9749250aec5a5af0 [id=0b1b2c3855f54d5c97bd2866dfbd8a5d]). Expected version 2 got 1
I20250114 02:54:09.227506 898 leader_election.cc:304] T 2a88342b30704cbda0f11f9ddc312fab P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
I20250114 02:54:09.228945 1211 raft_consensus.cc:2798] T 2a88342b30704cbda0f11f9ddc312fab P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:09.229416 1211 raft_consensus.cc:491] T 2a88342b30704cbda0f11f9ddc312fab P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:09.229849 1211 raft_consensus.cc:3054] T 2a88342b30704cbda0f11f9ddc312fab P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:09.231899 466 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_7dcbf016e9274d768c3d742c35ce24bd" } modify_external_catalogs: true
I20250114 02:54:09.232458 466 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_7dcbf016e9274d768c3d742c35ce24bd" } modify_external_catalogs: true
I20250114 02:54:09.233922 1104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a88342b30704cbda0f11f9ddc312fab" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:09.234583 1104 raft_consensus.cc:2463] T 2a88342b30704cbda0f11f9ddc312fab P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:09.243078 796 tablet_service.cc:1467] Processing CreateTablet for tablet ba074d4ea17747d5aa9287bdaf541681 (DEFAULT_TABLE table=default.Table_017604272128419e9c9d1bba85e595cb [id=253dc2a0f94149a78b0f4ab001fba488]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:09.243570 1258 tablet.cc:1721] T a1cf92e5275b454ca893b12f2d90d25b P 819f9e3412e54d44abb11d8b3ca877c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250114 02:54:09.244158 796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba074d4ea17747d5aa9287bdaf541681. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:09.246796 941 tablet_service.cc:1467] Processing CreateTablet for tablet ba074d4ea17747d5aa9287bdaf541681 (DEFAULT_TABLE table=default.Table_017604272128419e9c9d1bba85e595cb [id=253dc2a0f94149a78b0f4ab001fba488]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:09.246563 1257 tablet.cc:1721] T a1cf92e5275b454ca893b12f2d90d25b P 525198593c8e47afaee54e74267b87e3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250114 02:54:09.248155 941 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba074d4ea17747d5aa9287bdaf541681. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:09.249341 1084 tablet_service.cc:1467] Processing CreateTablet for tablet ba074d4ea17747d5aa9287bdaf541681 (DEFAULT_TABLE table=default.Table_017604272128419e9c9d1bba85e595cb [id=253dc2a0f94149a78b0f4ab001fba488]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:09.250380 1084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba074d4ea17747d5aa9287bdaf541681. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:09.260032 1083 tablet_service.cc:1467] Processing CreateTablet for tablet ec3285525faf482d8739f1bedd653503 (DEFAULT_TABLE table=default.Table_4598efdd6ff348d081ed8c3e7dc3130f [id=fdd50384f4894e20b21f109c6a33864c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:09.261229 1083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec3285525faf482d8739f1bedd653503. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:09.269369 1206 tablet_bootstrap.cc:492] T 54f9bb2d2e334f53b0c0560f26bbfa3f P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:09.269778 1206 ts_tablet_manager.cc:1397] T 54f9bb2d2e334f53b0c0560f26bbfa3f P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.109s user 0.010s sys 0.000s
I20250114 02:54:09.275383 798 tablet_service.cc:1467] Processing CreateTablet for tablet ec3285525faf482d8739f1bedd653503 (DEFAULT_TABLE table=default.Table_4598efdd6ff348d081ed8c3e7dc3130f [id=fdd50384f4894e20b21f109c6a33864c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:09.274958 1205 tablet_bootstrap.cc:654] T ee4ed8205f5641bf96b3f617d7f81321 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:09.276602 798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec3285525faf482d8739f1bedd653503. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:09.277129 1211 raft_consensus.cc:513] T 2a88342b30704cbda0f11f9ddc312fab P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:09.280416 795 tablet_service.cc:1467] Processing CreateTablet for tablet 180a3965833b4ac29467e5207e45d500 (DEFAULT_TABLE table=default.Table_97368d1825cf41e981b17a5e7c8d2148 [id=c82d594dc3d94b0981444ed5805e2034]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:09.281304 795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 180a3965833b4ac29467e5207e45d500. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:09.286132 1272 tablet.cc:1721] T a1cf92e5275b454ca893b12f2d90d25b P 90d6c2c58bb54e0eafb365625ac33871: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250114 02:54:09.271834 1206 raft_consensus.cc:357] T 54f9bb2d2e334f53b0c0560f26bbfa3f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:09.298547 1206 raft_consensus.cc:383] T 54f9bb2d2e334f53b0c0560f26bbfa3f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:09.298871 1206 raft_consensus.cc:738] T 54f9bb2d2e334f53b0c0560f26bbfa3f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:09.299482 1206 consensus_queue.cc:260] T 54f9bb2d2e334f53b0c0560f26bbfa3f P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:09.302614 1082 tablet_service.cc:1467] Processing CreateTablet for tablet 180a3965833b4ac29467e5207e45d500 (DEFAULT_TABLE table=default.Table_97368d1825cf41e981b17a5e7c8d2148 [id=c82d594dc3d94b0981444ed5805e2034]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:09.303823 1082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 180a3965833b4ac29467e5207e45d500. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:09.305651 1206 ts_tablet_manager.cc:1428] T 54f9bb2d2e334f53b0c0560f26bbfa3f P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.036s user 0.008s sys 0.000s
I20250114 02:54:09.285619 939 tablet_service.cc:1467] Processing CreateTablet for tablet ec3285525faf482d8739f1bedd653503 (DEFAULT_TABLE table=default.Table_4598efdd6ff348d081ed8c3e7dc3130f [id=fdd50384f4894e20b21f109c6a33864c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:09.307444 939 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec3285525faf482d8739f1bedd653503. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:09.286767 942 tablet_service.cc:1467] Processing CreateTablet for tablet 180a3965833b4ac29467e5207e45d500 (DEFAULT_TABLE table=default.Table_97368d1825cf41e981b17a5e7c8d2148 [id=c82d594dc3d94b0981444ed5805e2034]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:09.308964 942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 180a3965833b4ac29467e5207e45d500. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:09.311153 1206 tablet_bootstrap.cc:492] T a456420f78714cf895528e19dfe0ee8d P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:09.316293 470 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_3497e36f166142c3a1ef161324c8f45f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:09.318107 470 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3497e36f166142c3a1ef161324c8f45f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:09.325968 1205 tablet_bootstrap.cc:492] T ee4ed8205f5641bf96b3f617d7f81321 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:09.326362 1205 ts_tablet_manager.cc:1397] T ee4ed8205f5641bf96b3f617d7f81321 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.117s user 0.005s sys 0.004s
I20250114 02:54:09.326530 1258 tablet.cc:1721] T 2837990edc4d4bea871a8114533c9352 P 819f9e3412e54d44abb11d8b3ca877c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250114 02:54:09.329000 465 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_5b40399263bc473ca44553f9344f048a" } new_table_name: "default.Table_4720eed4e94c42f589ad0117109d5d77" modify_external_catalogs: true
I20250114 02:54:09.331560 1206 tablet_bootstrap.cc:654] T a456420f78714cf895528e19dfe0ee8d P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:09.333003 796 tablet_service.cc:1467] Processing CreateTablet for tablet 34b7a95953f043a08d1bce25d7af4c00 (DEFAULT_TABLE table=default.Table_d736c43978ca4363949e814cc78b057e [id=c14ed2074c144373a0314f99813e8da3]), partition=RANGE (key) PARTITION UNBOUNDED
W20250114 02:54:09.333303 457 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet ba074d4ea17747d5aa9287bdaf541681 (table default.Table_017604272128419e9c9d1bba85e595cb [id=253dc2a0f94149a78b0f4ab001fba488], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet ba074d4ea17747d5aa9287bdaf541681 consensus state has no leader
I20250114 02:54:09.334018 796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34b7a95953f043a08d1bce25d7af4c00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:09.335124 938 tablet_service.cc:1467] Processing CreateTablet for tablet 34b7a95953f043a08d1bce25d7af4c00 (DEFAULT_TABLE table=default.Table_d736c43978ca4363949e814cc78b057e [id=c14ed2074c144373a0314f99813e8da3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:09.336580 938 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34b7a95953f043a08d1bce25d7af4c00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:09.340601 1205 raft_consensus.cc:357] T ee4ed8205f5641bf96b3f617d7f81321 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:09.341253 1205 raft_consensus.cc:383] T ee4ed8205f5641bf96b3f617d7f81321 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:09.341486 1205 raft_consensus.cc:738] T ee4ed8205f5641bf96b3f617d7f81321 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:09.342914 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a88342b30704cbda0f11f9ddc312fab" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:09.343479 818 raft_consensus.cc:3054] T 2a88342b30704cbda0f11f9ddc312fab P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:09.343701 1205 consensus_queue.cc:260] T ee4ed8205f5641bf96b3f617d7f81321 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:09.345850 1222 ts_tablet_manager.cc:1918] T 9ded0f4f7b0244c28d56bde1c7ec93a6 P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:09.346184 1222 log.cc:1198] T 9ded0f4f7b0244c28d56bde1c7ec93a6 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/9ded0f4f7b0244c28d56bde1c7ec93a6
I20250114 02:54:09.346802 1222 ts_tablet_manager.cc:1939] T 9ded0f4f7b0244c28d56bde1c7ec93a6 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:09.349956 1205 ts_tablet_manager.cc:1428] T ee4ed8205f5641bf96b3f617d7f81321 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.023s user 0.006s sys 0.000s
I20250114 02:54:09.349931 1227 leader_election.cc:290] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Requested vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:09.350711 1205 tablet_bootstrap.cc:492] T a456420f78714cf895528e19dfe0ee8d P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:09.352087 1223 ts_tablet_manager.cc:1905] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:09.362865 464 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_bf49e6df9e43470da41cf3037638033a" } modify_external_catalogs: true
I20250114 02:54:09.363596 464 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_bf49e6df9e43470da41cf3037638033a" } modify_external_catalogs: true
I20250114 02:54:09.370388 1081 tablet_service.cc:1467] Processing CreateTablet for tablet 34b7a95953f043a08d1bce25d7af4c00 (DEFAULT_TABLE table=default.Table_d736c43978ca4363949e814cc78b057e [id=c14ed2074c144373a0314f99813e8da3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:09.371579 1081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34b7a95953f043a08d1bce25d7af4c00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:09.377537 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet 9ded0f4f7b0244c28d56bde1c7ec93a6 (table default.Table_a027831c951c4cf88c96d846f8cd360b [id=5c11573dbe494f24927ab32dcefded96]) successfully deleted
I20250114 02:54:09.378468 1222 tablet_replica.cc:331] T 54f9bb2d2e334f53b0c0560f26bbfa3f P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:09.382804 1222 raft_consensus.cc:2238] T 54f9bb2d2e334f53b0c0560f26bbfa3f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:09.383275 1222 raft_consensus.cc:2267] T 54f9bb2d2e334f53b0c0560f26bbfa3f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:09.383749 471 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_6e68b9b0d43c417cb8591764ef6044f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:09.388881 1259 tablet.cc:1721] T 2837990edc4d4bea871a8114533c9352 P 90d6c2c58bb54e0eafb365625ac33871: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250114 02:54:09.393417 1222 ts_tablet_manager.cc:1905] T 54f9bb2d2e334f53b0c0560f26bbfa3f P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250114 02:54:09.393347 471 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6e68b9b0d43c417cb8591764ef6044f5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:09.394065 1205 tablet_bootstrap.cc:654] T a456420f78714cf895528e19dfe0ee8d P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:09.402523 466 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet 6ba6181d7ee44a29a992a8d42f2b27f5
I20250114 02:54:09.402786 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a88342b30704cbda0f11f9ddc312fab" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:09.403424 1106 raft_consensus.cc:3054] T 2a88342b30704cbda0f11f9ddc312fab P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:09.400601 1206 tablet_bootstrap.cc:492] T a456420f78714cf895528e19dfe0ee8d P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:09.404525 1206 ts_tablet_manager.cc:1397] T a456420f78714cf895528e19dfe0ee8d P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.094s user 0.009s sys 0.000s
I20250114 02:54:09.405606 797 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ba6181d7ee44a29a992a8d42f2b27f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:09 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:09.407598 1206 raft_consensus.cc:357] T a456420f78714cf895528e19dfe0ee8d P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:09.411438 1106 raft_consensus.cc:2463] T 2a88342b30704cbda0f11f9ddc312fab P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:09.412525 898 leader_election.cc:304] T 2a88342b30704cbda0f11f9ddc312fab P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:09.413285 1249 raft_consensus.cc:2798] T 2a88342b30704cbda0f11f9ddc312fab P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:09.413684 1249 raft_consensus.cc:695] T 2a88342b30704cbda0f11f9ddc312fab P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Becoming Leader. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Running, Role: LEADER
I20250114 02:54:09.414407 1249 consensus_queue.cc:237] T 2a88342b30704cbda0f11f9ddc312fab P 525198593c8e47afaee54e74267b87e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:09.424296 1211 leader_election.cc:290] T 2a88342b30704cbda0f11f9ddc312fab P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Requested vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:09.424316 466 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_d736c43978ca4363949e814cc78b057e" } new_table_name: "default.Table_c99fdc20f32d41baadd11eb425b279ac" modify_external_catalogs: true
I20250114 02:54:09.428052 1206 raft_consensus.cc:383] T a456420f78714cf895528e19dfe0ee8d P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:09.428354 1206 raft_consensus.cc:738] T a456420f78714cf895528e19dfe0ee8d P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:09.428970 1206 consensus_queue.cc:260] T a456420f78714cf895528e19dfe0ee8d P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:09.432387 818 raft_consensus.cc:2463] T 2a88342b30704cbda0f11f9ddc312fab P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:09.434635 467 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: default.Table_63284ca278eb44d8b01f8e7c34a7a6e3 [id=9ec503e449fb4a2a96bbb77237c2fb4f] alter complete (version 3)
I20250114 02:54:09.438678 1207 tablet_bootstrap.cc:654] T 9ded0f4f7b0244c28d56bde1c7ec93a6 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
W20250114 02:54:09.438701 465 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a3886c5f976445cebc95ef1839a74aa9 has no consensus state
I20250114 02:54:09.439953 1206 ts_tablet_manager.cc:1428] T a456420f78714cf895528e19dfe0ee8d P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.035s user 0.006s sys 0.000s
I20250114 02:54:09.451839 1206 tablet_bootstrap.cc:492] T ee4ed8205f5641bf96b3f617d7f81321 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:09.478391 940 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ba6181d7ee44a29a992a8d42f2b27f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:09 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:09.479045 1225 tablet_replica.cc:331] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:09.479753 1225 raft_consensus.cc:2238] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Raft consensus shutting down.
I20250114 02:54:09.480686 1225 raft_consensus.cc:2267] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:09.480990 1205 tablet_bootstrap.cc:492] T a456420f78714cf895528e19dfe0ee8d P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:09.481336 1205 ts_tablet_manager.cc:1397] T a456420f78714cf895528e19dfe0ee8d P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.131s user 0.010s sys 0.005s
I20250114 02:54:09.482861 1225 ts_tablet_manager.cc:1905] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:09.484119 1205 raft_consensus.cc:357] T a456420f78714cf895528e19dfe0ee8d P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:09.484757 1205 raft_consensus.cc:383] T a456420f78714cf895528e19dfe0ee8d P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:09.485011 1205 raft_consensus.cc:738] T a456420f78714cf895528e19dfe0ee8d P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:09.485589 1205 consensus_queue.cc:260] T a456420f78714cf895528e19dfe0ee8d P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:09.472002 1222 ts_tablet_manager.cc:1918] T 54f9bb2d2e334f53b0c0560f26bbfa3f P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:09.490206 1205 ts_tablet_manager.cc:1428] T a456420f78714cf895528e19dfe0ee8d P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20250114 02:54:09.492558 1222 log.cc:1198] T 54f9bb2d2e334f53b0c0560f26bbfa3f P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/54f9bb2d2e334f53b0c0560f26bbfa3f
I20250114 02:54:09.493170 1222 ts_tablet_manager.cc:1939] T 54f9bb2d2e334f53b0c0560f26bbfa3f P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:09.493963 1205 tablet_bootstrap.cc:492] T ec3285525faf482d8739f1bedd653503 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:09.496665 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet 54f9bb2d2e334f53b0c0560f26bbfa3f (table default.Table_c6640103d84c48beb554d7509b246b27 [id=4873b55a952644e3919e7759a9c59b07]) successfully deleted
I20250114 02:54:09.500149 1257 tablet.cc:1721] T 2837990edc4d4bea871a8114533c9352 P 525198593c8e47afaee54e74267b87e3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250114 02:54:09.504555 1206 tablet_bootstrap.cc:654] T ee4ed8205f5641bf96b3f617d7f81321 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:09.515692 467 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_e691c983a1bc4ce3a3d4ac683c048218"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:09.502450 1205 tablet_bootstrap.cc:654] T ec3285525faf482d8739f1bedd653503 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
W20250114 02:54:09.517547 467 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e691c983a1bc4ce3a3d4ac683c048218 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:09.518409 1223 ts_tablet_manager.cc:1918] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:09.518602 470 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_901e963b101d4f13b2926c99a4c3c9cb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:09.518914 1223 log.cc:1198] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/8b27d06a25264f2ebc6f6cc3dd0a2c07
I20250114 02:54:09.519795 1223 ts_tablet_manager.cc:1939] T 8b27d06a25264f2ebc6f6cc3dd0a2c07 P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
W20250114 02:54:09.520082 470 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_901e963b101d4f13b2926c99a4c3c9cb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:09.508322 1222 tablet_replica.cc:331] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:09.515403 1212 raft_consensus.cc:491] T a456420f78714cf895528e19dfe0ee8d P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:09.532054 1212 raft_consensus.cc:513] T a456420f78714cf895528e19dfe0ee8d P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:09.532552 1222 raft_consensus.cc:2238] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:09.533344 1222 raft_consensus.cc:2267] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:09.533906 1212 leader_election.cc:290] T a456420f78714cf895528e19dfe0ee8d P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:09.536257 1222 ts_tablet_manager.cc:1905] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:09.537781 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a456420f78714cf895528e19dfe0ee8d" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:09.538472 962 raft_consensus.cc:2463] T a456420f78714cf895528e19dfe0ee8d P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 0.
I20250114 02:54:09.539490 751 leader_election.cc:304] T a456420f78714cf895528e19dfe0ee8d P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
I20250114 02:54:09.540112 1205 tablet_bootstrap.cc:492] T ec3285525faf482d8739f1bedd653503 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:09.540279 1212 raft_consensus.cc:2798] T a456420f78714cf895528e19dfe0ee8d P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:09.540561 1205 ts_tablet_manager.cc:1397] T ec3285525faf482d8739f1bedd653503 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.047s user 0.009s sys 0.002s
I20250114 02:54:09.540570 1212 raft_consensus.cc:491] T a456420f78714cf895528e19dfe0ee8d P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:09.540846 1212 raft_consensus.cc:3054] T a456420f78714cf895528e19dfe0ee8d P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:09.542691 1205 raft_consensus.cc:357] T ec3285525faf482d8739f1bedd653503 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:09.546530 1205 raft_consensus.cc:383] T ec3285525faf482d8739f1bedd653503 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:09.546926 1205 raft_consensus.cc:738] T ec3285525faf482d8739f1bedd653503 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:09.546711 1212 raft_consensus.cc:513] T a456420f78714cf895528e19dfe0ee8d P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:09.548231 1205 consensus_queue.cc:260] T ec3285525faf482d8739f1bedd653503 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:09.549496 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet 8b27d06a25264f2ebc6f6cc3dd0a2c07 (table default.Table_480725b9282e4dd7a13623a6dab4f9ee [id=a6c456511d9748f7b2c1d95ae7382f5d]) successfully deleted
I20250114 02:54:09.550297 1205 ts_tablet_manager.cc:1428] T ec3285525faf482d8739f1bedd653503 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
W20250114 02:54:09.550911 458 catalog_manager.cc:4852] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 54f9bb2d2e334f53b0c0560f26bbfa3f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 54f9bb2d2e334f53b0c0560f26bbfa3f already in progress: creating tablet
I20250114 02:54:09.552316 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a456420f78714cf895528e19dfe0ee8d" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:09.553701 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a456420f78714cf895528e19dfe0ee8d" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
W20250114 02:54:09.554729 753 leader_election.cc:343] T a456420f78714cf895528e19dfe0ee8d P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
W20250114 02:54:09.555768 753 leader_election.cc:343] T a456420f78714cf895528e19dfe0ee8d P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:09.557265 1085 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ba6181d7ee44a29a992a8d42f2b27f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:09 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:09.564657 1206 tablet_bootstrap.cc:492] T ee4ed8205f5641bf96b3f617d7f81321 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:09.565044 1206 ts_tablet_manager.cc:1397] T ee4ed8205f5641bf96b3f617d7f81321 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.113s user 0.010s sys 0.000s
I20250114 02:54:09.566421 1084 tablet_service.cc:1467] Processing CreateTablet for tablet c69d5eef3f3f49699e76dcbe1be7d6f6 (DEFAULT_TABLE table=default.Table_24dc0a51cadc4387918d43a65986ebf4 [id=0661845e06704345b8498b0060e3a40d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:09.566969 796 tablet_service.cc:1467] Processing CreateTablet for tablet 857119ca5f9d4ad98f899df22f32244b (DEFAULT_TABLE table=default.Table_55dad15f1b4f4e22854ba3070b0eea8e [id=5f5b8d19482b4137add134b196bfd20d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:09.567487 1084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c69d5eef3f3f49699e76dcbe1be7d6f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:09.568007 796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 857119ca5f9d4ad98f899df22f32244b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:09.570745 797 tablet_service.cc:1467] Processing CreateTablet for tablet c69d5eef3f3f49699e76dcbe1be7d6f6 (DEFAULT_TABLE table=default.Table_24dc0a51cadc4387918d43a65986ebf4 [id=0661845e06704345b8498b0060e3a40d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:09.570767 794 tablet_service.cc:1467] Processing CreateTablet for tablet 96420dece19549b3876bd795cbdc0f0a (DEFAULT_TABLE table=default.Table_d12c2d9a4b3f42d4839548ca62645693 [id=ceeff15df8714e78b662b1cb4e4fc105]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:09.572296 794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96420dece19549b3876bd795cbdc0f0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:09.574249 793 tablet_service.cc:1467] Processing CreateTablet for tablet a4dd92b0133e413d8b895a0f33e2faf1 (DEFAULT_TABLE table=default.Table_075cadc0bfa145718aa9f2f6850f595c [id=ec0fc676edc842b2b7114a4b3fe0626a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:09.574519 462 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet 2837990edc4d4bea871a8114533c9352 (table default.Table_1f95cb34af29429a9749250aec5a5af0 [id=0b1b2c3855f54d5c97bd2866dfbd8a5d]). Expected version 2 got 1
I20250114 02:54:09.575366 793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4dd92b0133e413d8b895a0f33e2faf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:09.576237 797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c69d5eef3f3f49699e76dcbe1be7d6f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:09.584976 939 tablet_service.cc:1467] Processing CreateTablet for tablet c69d5eef3f3f49699e76dcbe1be7d6f6 (DEFAULT_TABLE table=default.Table_24dc0a51cadc4387918d43a65986ebf4 [id=0661845e06704345b8498b0060e3a40d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:09.585662 937 tablet_service.cc:1467] Processing CreateTablet for tablet a4dd92b0133e413d8b895a0f33e2faf1 (DEFAULT_TABLE table=default.Table_075cadc0bfa145718aa9f2f6850f595c [id=ec0fc676edc842b2b7114a4b3fe0626a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:09.586086 939 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c69d5eef3f3f49699e76dcbe1be7d6f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:09.587754 937 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4dd92b0133e413d8b895a0f33e2faf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:09.592988 940 tablet_service.cc:1467] Processing CreateTablet for tablet 96420dece19549b3876bd795cbdc0f0a (DEFAULT_TABLE table=default.Table_d12c2d9a4b3f42d4839548ca62645693 [id=ceeff15df8714e78b662b1cb4e4fc105]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:09.594045 940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96420dece19549b3876bd795cbdc0f0a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250114 02:54:09.597522 466 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 34b7a95953f043a08d1bce25d7af4c00 consensus state has no leader
I20250114 02:54:09.583768 1085 tablet_service.cc:1467] Processing CreateTablet for tablet 857119ca5f9d4ad98f899df22f32244b (DEFAULT_TABLE table=default.Table_55dad15f1b4f4e22854ba3070b0eea8e [id=5f5b8d19482b4137add134b196bfd20d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:09.585529 1081 tablet_service.cc:1467] Processing CreateTablet for tablet a4dd92b0133e413d8b895a0f33e2faf1 (DEFAULT_TABLE table=default.Table_075cadc0bfa145718aa9f2f6850f595c [id=ec0fc676edc842b2b7114a4b3fe0626a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:09.601125 1081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4dd92b0133e413d8b895a0f33e2faf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:09.596971 792 tablet_service.cc:1467] Processing CreateTablet for tablet a3886c5f976445cebc95ef1839a74aa9 (DEFAULT_TABLE table=default.Table_4720eed4e94c42f589ad0117109d5d77 [id=e984149e9da444f1b2802e3f35a60fd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:09.587015 1086 tablet_service.cc:1467] Processing CreateTablet for tablet a3886c5f976445cebc95ef1839a74aa9 (DEFAULT_TABLE table=default.Table_4720eed4e94c42f589ad0117109d5d77 [id=e984149e9da444f1b2802e3f35a60fd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:09.604590 792 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3886c5f976445cebc95ef1839a74aa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:09.604900 1086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3886c5f976445cebc95ef1839a74aa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:09.605665 941 tablet_service.cc:1467] Processing CreateTablet for tablet 857119ca5f9d4ad98f899df22f32244b (DEFAULT_TABLE table=default.Table_55dad15f1b4f4e22854ba3070b0eea8e [id=5f5b8d19482b4137add134b196bfd20d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:09.606638 941 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 857119ca5f9d4ad98f899df22f32244b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:09.588721 1083 tablet_service.cc:1467] Processing CreateTablet for tablet 96420dece19549b3876bd795cbdc0f0a (DEFAULT_TABLE table=default.Table_d12c2d9a4b3f42d4839548ca62645693 [id=ceeff15df8714e78b662b1cb4e4fc105]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:09.591082 1207 tablet_bootstrap.cc:492] T 9ded0f4f7b0244c28d56bde1c7ec93a6 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:09.609053 1207 ts_tablet_manager.cc:1397] T 9ded0f4f7b0244c28d56bde1c7ec93a6 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.395s user 0.008s sys 0.005s
I20250114 02:54:09.609426 1083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96420dece19549b3876bd795cbdc0f0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:09.612685 466 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_d12c2d9a4b3f42d4839548ca62645693" } modify_external_catalogs: true
I20250114 02:54:09.613317 466 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_d12c2d9a4b3f42d4839548ca62645693" } modify_external_catalogs: true
I20250114 02:54:09.613946 1207 raft_consensus.cc:357] T 9ded0f4f7b0244c28d56bde1c7ec93a6 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:09.614627 1207 raft_consensus.cc:383] T 9ded0f4f7b0244c28d56bde1c7ec93a6 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:09.614879 1207 raft_consensus.cc:738] T 9ded0f4f7b0244c28d56bde1c7ec93a6 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:09.615478 1207 consensus_queue.cc:260] T 9ded0f4f7b0244c28d56bde1c7ec93a6 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:09.616604 936 tablet_service.cc:1467] Processing CreateTablet for tablet a3886c5f976445cebc95ef1839a74aa9 (DEFAULT_TABLE table=default.Table_4720eed4e94c42f589ad0117109d5d77 [id=e984149e9da444f1b2802e3f35a60fd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:09.617676 936 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3886c5f976445cebc95ef1839a74aa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:09.624531 1205 tablet_bootstrap.cc:492] T ba074d4ea17747d5aa9287bdaf541681 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:09.643419 471 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_8a880e0ae5ff4b5d8de8f40f15dd6e02"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:09.644969 1085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 857119ca5f9d4ad98f899df22f32244b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250114 02:54:09.645105 471 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8a880e0ae5ff4b5d8de8f40f15dd6e02 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:09.645229 1207 ts_tablet_manager.cc:1428] T 9ded0f4f7b0244c28d56bde1c7ec93a6 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.036s user 0.005s sys 0.000s
I20250114 02:54:09.644805 1206 raft_consensus.cc:357] T ee4ed8205f5641bf96b3f617d7f81321 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:09.645531 1206 raft_consensus.cc:383] T ee4ed8205f5641bf96b3f617d7f81321 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:09.645767 1206 raft_consensus.cc:738] T ee4ed8205f5641bf96b3f617d7f81321 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:09.646059 1207 tablet_bootstrap.cc:492] T 54f9bb2d2e334f53b0c0560f26bbfa3f P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:09.646351 1206 consensus_queue.cc:260] T ee4ed8205f5641bf96b3f617d7f81321 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:09.646799 1227 raft_consensus.cc:491] T 9ded0f4f7b0244c28d56bde1c7ec93a6 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:09.647217 1227 raft_consensus.cc:513] T 9ded0f4f7b0244c28d56bde1c7ec93a6 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:09.647743 1212 leader_election.cc:290] T a456420f78714cf895528e19dfe0ee8d P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Requested vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:09.650779 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a456420f78714cf895528e19dfe0ee8d" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
I20250114 02:54:09.651353 962 raft_consensus.cc:3054] T a456420f78714cf895528e19dfe0ee8d P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:09.656059 468 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_1f95cb34af29429a9749250aec5a5af0" } new_table_name: "default.Table_ba8866c1c90041268e09ff59a46503f1" modify_external_catalogs: true
W20250114 02:54:09.666291 1042 leader_election.cc:343] T 9ded0f4f7b0244c28d56bde1c7ec93a6 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Not found: Tablet not found: 9ded0f4f7b0244c28d56bde1c7ec93a6
I20250114 02:54:09.667225 1227 leader_election.cc:290] T 9ded0f4f7b0244c28d56bde1c7ec93a6 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:09.668864 1223 tablet_replica.cc:331] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:09.668792 1222 ts_tablet_manager.cc:1918] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250114 02:54:09.669243 1222 log.cc:1198] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/6ba6181d7ee44a29a992a8d42f2b27f5
I20250114 02:54:09.670058 1222 ts_tablet_manager.cc:1939] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:09.672402 1223 raft_consensus.cc:2238] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:09.676681 962 raft_consensus.cc:2463] T a456420f78714cf895528e19dfe0ee8d P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 1.
I20250114 02:54:09.678778 1223 raft_consensus.cc:2267] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250114 02:54:09.671506 1040 leader_election.cc:343] T 9ded0f4f7b0244c28d56bde1c7ec93a6 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Not found: Tablet not found: 9ded0f4f7b0244c28d56bde1c7ec93a6
I20250114 02:54:09.679610 1040 leader_election.cc:304] T 9ded0f4f7b0244c28d56bde1c7ec93a6 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90d6c2c58bb54e0eafb365625ac33871; no voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2
I20250114 02:54:09.681525 1223 ts_tablet_manager.cc:1905] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:09.682204 1227 raft_consensus.cc:2743] T 9ded0f4f7b0244c28d56bde1c7ec93a6 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:09.691676 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet 6ba6181d7ee44a29a992a8d42f2b27f5 (table default.Table_7dcbf016e9274d768c3d742c35ce24bd [id=e707381ee3f24b6bb3275bab2cf8fc38]) successfully deleted
I20250114 02:54:09.692461 1206 ts_tablet_manager.cc:1428] T ee4ed8205f5641bf96b3f617d7f81321 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.127s user 0.007s sys 0.000s
I20250114 02:54:09.678682 1225 ts_tablet_manager.cc:1918] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250114 02:54:09.692909 1225 log.cc:1198] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/6ba6181d7ee44a29a992a8d42f2b27f5
I20250114 02:54:09.693782 1225 ts_tablet_manager.cc:1939] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:09.695953 1206 tablet_bootstrap.cc:492] T ba074d4ea17747d5aa9287bdaf541681 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:09.696143 751 leader_election.cc:304] T a456420f78714cf895528e19dfe0ee8d P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters: 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:09.704380 1294 raft_consensus.cc:2798] T a456420f78714cf895528e19dfe0ee8d P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:09.704934 1294 raft_consensus.cc:695] T a456420f78714cf895528e19dfe0ee8d P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 LEADER]: Becoming Leader. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Running, Role: LEADER
I20250114 02:54:09.705797 1294 consensus_queue.cc:237] T a456420f78714cf895528e19dfe0ee8d P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:09.733289 1207 tablet_bootstrap.cc:654] T 54f9bb2d2e334f53b0c0560f26bbfa3f P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:09.733312 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet 6ba6181d7ee44a29a992a8d42f2b27f5 (table default.Table_7dcbf016e9274d768c3d742c35ce24bd [id=e707381ee3f24b6bb3275bab2cf8fc38]) successfully deleted
I20250114 02:54:09.751369 467 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_2842b4625fde43f4b83c5cf150c6ae93"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:09.753021 467 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2842b4625fde43f4b83c5cf150c6ae93 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:09.760532 464 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet ee4ed8205f5641bf96b3f617d7f81321
I20250114 02:54:09.762645 1084 tablet_service.cc:1514] Processing DeleteTablet for tablet ee4ed8205f5641bf96b3f617d7f81321 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:09 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:09.765609 793 tablet_service.cc:1514] Processing DeleteTablet for tablet ee4ed8205f5641bf96b3f617d7f81321 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:09 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:09.766469 1222 tablet_replica.cc:331] T ee4ed8205f5641bf96b3f617d7f81321 P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:09.767277 1222 raft_consensus.cc:2238] T ee4ed8205f5641bf96b3f617d7f81321 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:09.767730 1222 raft_consensus.cc:2267] T ee4ed8205f5641bf96b3f617d7f81321 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:09.768347 1206 tablet_bootstrap.cc:654] T ba074d4ea17747d5aa9287bdaf541681 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:09.770130 1222 ts_tablet_manager.cc:1905] T ee4ed8205f5641bf96b3f617d7f81321 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:09.782045 1205 tablet_bootstrap.cc:654] T ba074d4ea17747d5aa9287bdaf541681 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:09.785969 939 tablet_service.cc:1514] Processing DeleteTablet for tablet ee4ed8205f5641bf96b3f617d7f81321 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:09 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:09.786566 938 tablet_service.cc:1514] Processing DeleteTablet for tablet 54f9bb2d2e334f53b0c0560f26bbfa3f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:09 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:09.786777 1225 tablet_replica.cc:331] T ee4ed8205f5641bf96b3f617d7f81321 P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:09.787592 1225 raft_consensus.cc:2238] T ee4ed8205f5641bf96b3f617d7f81321 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:09.788057 1225 raft_consensus.cc:2267] T ee4ed8205f5641bf96b3f617d7f81321 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:09.790354 1225 ts_tablet_manager.cc:1905] T ee4ed8205f5641bf96b3f617d7f81321 P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250114 02:54:09.792337 455 consensus_peers.cc:487] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750 -> Peer 87b3db09ac87436e9a60d6de5fedaff5 (127.0.107.124:39105): Couldn't send request to peer 87b3db09ac87436e9a60d6de5fedaff5. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.124:39105: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250114 02:54:09.800740 464 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_901e963b101d4f13b2926c99a4c3c9cb" } new_table_name: "default.Table_ddecd244bb3d44368792bd942634b1c9" modify_external_catalogs: true
I20250114 02:54:09.806242 1206 tablet_bootstrap.cc:492] T ba074d4ea17747d5aa9287bdaf541681 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:09.806629 1206 ts_tablet_manager.cc:1397] T ba074d4ea17747d5aa9287bdaf541681 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.111s user 0.009s sys 0.000s
I20250114 02:54:09.809827 1206 raft_consensus.cc:357] T ba074d4ea17747d5aa9287bdaf541681 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:09.810453 1206 raft_consensus.cc:383] T ba074d4ea17747d5aa9287bdaf541681 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:09.810683 1206 raft_consensus.cc:738] T ba074d4ea17747d5aa9287bdaf541681 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:09.811298 1206 consensus_queue.cc:260] T ba074d4ea17747d5aa9287bdaf541681 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:09.813539 1222 ts_tablet_manager.cc:1918] T ee4ed8205f5641bf96b3f617d7f81321 P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:09.821990 1223 ts_tablet_manager.cc:1918] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250114 02:54:09.822448 1223 log.cc:1198] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/6ba6181d7ee44a29a992a8d42f2b27f5
I20250114 02:54:09.823186 1223 ts_tablet_manager.cc:1939] T 6ba6181d7ee44a29a992a8d42f2b27f5 P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
I20250114 02:54:09.825749 1207 tablet_bootstrap.cc:492] T 54f9bb2d2e334f53b0c0560f26bbfa3f P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:09.826156 1207 ts_tablet_manager.cc:1397] T 54f9bb2d2e334f53b0c0560f26bbfa3f P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.180s user 0.014s sys 0.000s
I20250114 02:54:09.829639 1207 raft_consensus.cc:357] T 54f9bb2d2e334f53b0c0560f26bbfa3f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:09.830839 1207 raft_consensus.cc:383] T 54f9bb2d2e334f53b0c0560f26bbfa3f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:09.838735 469 catalog_manager.cc:5526] T 2a88342b30704cbda0f11f9ddc312fab P 525198593c8e47afaee54e74267b87e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 525198593c8e47afaee54e74267b87e3 (127.0.107.66). New cstate: current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:09.843278 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet 6ba6181d7ee44a29a992a8d42f2b27f5 (table default.Table_7dcbf016e9274d768c3d742c35ce24bd [id=e707381ee3f24b6bb3275bab2cf8fc38]) successfully deleted
W20250114 02:54:09.843972 458 catalog_manager.cc:4852] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet ee4ed8205f5641bf96b3f617d7f81321 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee4ed8205f5641bf96b3f617d7f81321 already in progress: creating tablet
I20250114 02:54:09.845589 796 tablet_service.cc:1467] Processing CreateTablet for tablet d434cbc9daf549a6bdc04c8c76ead8a6 (DEFAULT_TABLE table=default.Table_6e68b9b0d43c417cb8591764ef6044f5 [id=5e6aa799d6444bad968f29327ebace9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:09.845888 1222 log.cc:1198] T ee4ed8205f5641bf96b3f617d7f81321 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/ee4ed8205f5641bf96b3f617d7f81321
I20250114 02:54:09.846668 796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d434cbc9daf549a6bdc04c8c76ead8a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:09.847294 1222 ts_tablet_manager.cc:1939] T ee4ed8205f5641bf96b3f617d7f81321 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:09.814846 1206 ts_tablet_manager.cc:1428] T ba074d4ea17747d5aa9287bdaf541681 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20250114 02:54:09.834422 793 tablet_service.cc:1467] Processing CreateTablet for tablet 92b8d48880434ab99eec7f1e452931fa (DEFAULT_TABLE table=default.Table_3497e36f166142c3a1ef161324c8f45f [id=a3a41d4e0b004a4aaefc1c1009a8f244]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:09.850140 793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92b8d48880434ab99eec7f1e452931fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:09.856961 1206 tablet_bootstrap.cc:492] T 34b7a95953f043a08d1bce25d7af4c00 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
W20250114 02:54:09.859084 464 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b66cbf624f3b4078b5102d1db657e67f has no consensus state
I20250114 02:54:09.868091 1249 raft_consensus.cc:491] T ec3285525faf482d8739f1bedd653503 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:09.868634 1249 raft_consensus.cc:513] T ec3285525faf482d8739f1bedd653503 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:09.869479 938 tablet_service.cc:1467] Processing CreateTablet for tablet d434cbc9daf549a6bdc04c8c76ead8a6 (DEFAULT_TABLE table=default.Table_6e68b9b0d43c417cb8591764ef6044f5 [id=5e6aa799d6444bad968f29327ebace9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:09.870497 938 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d434cbc9daf549a6bdc04c8c76ead8a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:09.870530 1249 leader_election.cc:290] T ec3285525faf482d8739f1bedd653503 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:09.844504 1085 tablet_service.cc:1467] Processing CreateTablet for tablet d434cbc9daf549a6bdc04c8c76ead8a6 (DEFAULT_TABLE table=default.Table_6e68b9b0d43c417cb8591764ef6044f5 [id=5e6aa799d6444bad968f29327ebace9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:09.844828 1084 tablet_service.cc:1467] Processing CreateTablet for tablet 92b8d48880434ab99eec7f1e452931fa (DEFAULT_TABLE table=default.Table_3497e36f166142c3a1ef161324c8f45f [id=a3a41d4e0b004a4aaefc1c1009a8f244]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:09.875330 1085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d434cbc9daf549a6bdc04c8c76ead8a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:09.875983 1084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92b8d48880434ab99eec7f1e452931fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:09.877044 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec3285525faf482d8739f1bedd653503" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:09.875780 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec3285525faf482d8739f1bedd653503" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:09.884562 1081 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ded0f4f7b0244c28d56bde1c7ec93a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:08 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:09.885242 468 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_7410fb88d36043feb97e96981c09d34e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:09.886785 898 leader_election.cc:343] T ec3285525faf482d8739f1bedd653503 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Illegal state: must be running to vote when last-logged opid is not known
W20250114 02:54:09.887459 468 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7410fb88d36043feb97e96981c09d34e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250114 02:54:09.888362 898 leader_election.cc:343] T ec3285525faf482d8739f1bedd653503 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:09.888891 898 leader_election.cc:304] T ec3285525faf482d8739f1bedd653503 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 525198593c8e47afaee54e74267b87e3; no voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:09.891027 937 tablet_service.cc:1467] Processing CreateTablet for tablet 92b8d48880434ab99eec7f1e452931fa (DEFAULT_TABLE table=default.Table_3497e36f166142c3a1ef161324c8f45f [id=a3a41d4e0b004a4aaefc1c1009a8f244]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:09.891494 466 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet 96420dece19549b3876bd795cbdc0f0a
I20250114 02:54:09.894160 939 tablet_service.cc:1514] Processing DeleteTablet for tablet 96420dece19549b3876bd795cbdc0f0a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:09 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:09.895888 1081 tablet_service.cc:1514] Processing DeleteTablet for tablet 96420dece19549b3876bd795cbdc0f0a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:09 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:09.896776 792 tablet_service.cc:1514] Processing DeleteTablet for tablet 54f9bb2d2e334f53b0c0560f26bbfa3f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:09 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:09.897871 792 tablet_service.cc:1514] Processing DeleteTablet for tablet 96420dece19549b3876bd795cbdc0f0a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:09 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:09.898641 1223 tablet_replica.cc:331] T 9ded0f4f7b0244c28d56bde1c7ec93a6 P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:09.902896 1206 tablet_bootstrap.cc:654] T 34b7a95953f043a08d1bce25d7af4c00 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:09.874871 1207 raft_consensus.cc:738] T 54f9bb2d2e334f53b0c0560f26bbfa3f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:09.903478 937 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92b8d48880434ab99eec7f1e452931fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:09.903975 1207 consensus_queue.cc:260] T 54f9bb2d2e334f53b0c0560f26bbfa3f P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:09.904124 1223 raft_consensus.cc:2238] T 9ded0f4f7b0244c28d56bde1c7ec93a6 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:09.905519 1223 raft_consensus.cc:2267] T 9ded0f4f7b0244c28d56bde1c7ec93a6 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:09.907740 465 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_7a318166ed3045c7bbae5b779be062c4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:09.909566 465 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7a318166ed3045c7bbae5b779be062c4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:09.910311 1207 ts_tablet_manager.cc:1428] T 54f9bb2d2e334f53b0c0560f26bbfa3f P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.084s user 0.007s sys 0.000s
I20250114 02:54:09.911995 1223 ts_tablet_manager.cc:1905] T 9ded0f4f7b0244c28d56bde1c7ec93a6 P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:09.916810 1235 raft_consensus.cc:2743] T ec3285525faf482d8739f1bedd653503 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:09.922551 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet ee4ed8205f5641bf96b3f617d7f81321 (table default.Table_bf49e6df9e43470da41cf3037638033a [id=e8c1f073ed504963947cab370eb9cf1a]) successfully deleted
I20250114 02:54:09.923218 1207 tablet_bootstrap.cc:492] T ee4ed8205f5641bf96b3f617d7f81321 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
W20250114 02:54:09.923254 458 catalog_manager.cc:4839] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet 54f9bb2d2e334f53b0c0560f26bbfa3f because the tablet was not found. No further retry: Not found: Tablet not found: 54f9bb2d2e334f53b0c0560f26bbfa3f
W20250114 02:54:09.923864 458 catalog_manager.cc:4852] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet 96420dece19549b3876bd795cbdc0f0a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 96420dece19549b3876bd795cbdc0f0a already in progress: creating tablet
I20250114 02:54:09.944128 1207 tablet_bootstrap.cc:654] T ee4ed8205f5641bf96b3f617d7f81321 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:09.944089 1225 ts_tablet_manager.cc:1918] T ee4ed8205f5641bf96b3f617d7f81321 P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:09.944595 1225 log.cc:1198] T ee4ed8205f5641bf96b3f617d7f81321 P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/ee4ed8205f5641bf96b3f617d7f81321
I20250114 02:54:09.945370 1225 ts_tablet_manager.cc:1939] T ee4ed8205f5641bf96b3f617d7f81321 P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:09.947141 1085 tablet_service.cc:1514] Processing DeleteTablet for tablet 54f9bb2d2e334f53b0c0560f26bbfa3f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:09 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:09.960915 466 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet ba074d4ea17747d5aa9287bdaf541681 (table default.Table_017604272128419e9c9d1bba85e595cb [id=253dc2a0f94149a78b0f4ab001fba488]). Expected version 1 got 0
I20250114 02:54:09.953989 1205 tablet_bootstrap.cc:492] T ba074d4ea17747d5aa9287bdaf541681 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:09.988204 1205 ts_tablet_manager.cc:1397] T ba074d4ea17747d5aa9287bdaf541681 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.364s user 0.014s sys 0.009s
I20250114 02:54:09.984973 465 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_ddecd244bb3d44368792bd942634b1c9" } modify_external_catalogs: true
I20250114 02:54:09.991482 465 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_ddecd244bb3d44368792bd942634b1c9" } modify_external_catalogs: true
I20250114 02:54:09.991782 466 catalog_manager.cc:5526] T a456420f78714cf895528e19dfe0ee8d P 819f9e3412e54d44abb11d8b3ca877c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65). New cstate: current_term: 1 leader_uuid: "819f9e3412e54d44abb11d8b3ca877c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:09.994170 1223 ts_tablet_manager.cc:1918] T 9ded0f4f7b0244c28d56bde1c7ec93a6 P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:09.994632 1223 log.cc:1198] T 9ded0f4f7b0244c28d56bde1c7ec93a6 P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/9ded0f4f7b0244c28d56bde1c7ec93a6
I20250114 02:54:09.995465 1223 ts_tablet_manager.cc:1939] T 9ded0f4f7b0244c28d56bde1c7ec93a6 P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
I20250114 02:54:09.986913 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet ee4ed8205f5641bf96b3f617d7f81321 (table default.Table_bf49e6df9e43470da41cf3037638033a [id=e8c1f073ed504963947cab370eb9cf1a]) successfully deleted
I20250114 02:54:09.998302 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet 9ded0f4f7b0244c28d56bde1c7ec93a6 (table default.Table_a027831c951c4cf88c96d846f8cd360b [id=5c11573dbe494f24927ab32dcefded96]) successfully deleted
I20250114 02:54:09.998689 1206 tablet_bootstrap.cc:492] T 34b7a95953f043a08d1bce25d7af4c00 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:09.999157 1206 ts_tablet_manager.cc:1397] T 34b7a95953f043a08d1bce25d7af4c00 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.142s user 0.009s sys 0.001s
W20250114 02:54:10.001420 458 catalog_manager.cc:4852] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 96420dece19549b3876bd795cbdc0f0a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 96420dece19549b3876bd795cbdc0f0a already in progress: creating tablet
I20250114 02:54:10.006170 1207 tablet_bootstrap.cc:492] T ee4ed8205f5641bf96b3f617d7f81321 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:10.006577 1207 ts_tablet_manager.cc:1397] T ee4ed8205f5641bf96b3f617d7f81321 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.084s user 0.016s sys 0.000s
I20250114 02:54:10.006364 469 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_2a1287091fb9466d9e2fae14499a0963"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:10.007478 1225 tablet_replica.cc:331] T 54f9bb2d2e334f53b0c0560f26bbfa3f P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:10.007472 468 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_b5367d1b4b2a4ec5a97350612e373c40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:10.008106 469 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2a1287091fb9466d9e2fae14499a0963 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:10.008347 1225 raft_consensus.cc:2238] T 54f9bb2d2e334f53b0c0560f26bbfa3f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:10.008817 1225 raft_consensus.cc:2267] T 54f9bb2d2e334f53b0c0560f26bbfa3f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:10.009150 1207 raft_consensus.cc:357] T ee4ed8205f5641bf96b3f617d7f81321 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:10.009889 1207 raft_consensus.cc:383] T ee4ed8205f5641bf96b3f617d7f81321 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:10.010144 1207 raft_consensus.cc:738] T ee4ed8205f5641bf96b3f617d7f81321 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:10.010664 1225 ts_tablet_manager.cc:1905] T 54f9bb2d2e334f53b0c0560f26bbfa3f P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:10.010723 1207 consensus_queue.cc:260] T ee4ed8205f5641bf96b3f617d7f81321 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:10.006546 1223 tablet_replica.cc:331] T 54f9bb2d2e334f53b0c0560f26bbfa3f P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:10.012426 1223 raft_consensus.cc:2238] T 54f9bb2d2e334f53b0c0560f26bbfa3f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:09.991892 1205 raft_consensus.cc:357] T ba074d4ea17747d5aa9287bdaf541681 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:10.012960 1205 raft_consensus.cc:383] T ba074d4ea17747d5aa9287bdaf541681 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:10.012928 1223 raft_consensus.cc:2267] T 54f9bb2d2e334f53b0c0560f26bbfa3f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:10.013229 1205 raft_consensus.cc:738] T ba074d4ea17747d5aa9287bdaf541681 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:10.015650 1223 ts_tablet_manager.cc:1905] T 54f9bb2d2e334f53b0c0560f26bbfa3f P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:10.026149 1205 consensus_queue.cc:260] T ba074d4ea17747d5aa9287bdaf541681 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:10.030313 465 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet b66cbf624f3b4078b5102d1db657e67f
I20250114 02:54:10.032521 1206 raft_consensus.cc:357] T 34b7a95953f043a08d1bce25d7af4c00 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:10.033216 1206 raft_consensus.cc:383] T 34b7a95953f043a08d1bce25d7af4c00 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:10.033473 1206 raft_consensus.cc:738] T 34b7a95953f043a08d1bce25d7af4c00 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:10.034036 1206 consensus_queue.cc:260] T 34b7a95953f043a08d1bce25d7af4c00 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:10.026990 1207 ts_tablet_manager.cc:1428] T ee4ed8205f5641bf96b3f617d7f81321 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.020s user 0.007s sys 0.000s
I20250114 02:54:10.040378 1207 tablet_bootstrap.cc:492] T 34b7a95953f043a08d1bce25d7af4c00 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:10.041962 1205 ts_tablet_manager.cc:1428] T ba074d4ea17747d5aa9287bdaf541681 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.053s user 0.007s sys 0.000s
I20250114 02:54:10.045540 1259 tablet.cc:1721] T 2837990edc4d4bea871a8114533c9352 P 90d6c2c58bb54e0eafb365625ac33871: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250114 02:54:10.054340 1084 tablet_service.cc:1514] Processing DeleteTablet for tablet ee4ed8205f5641bf96b3f617d7f81321 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:09 UTC) from {username='slave'} at 127.0.0.1:53876
W20250114 02:54:10.039283 468 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b5367d1b4b2a4ec5a97350612e373c40 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:10.065554 462 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_7410fb88d36043feb97e96981c09d34e" } new_table_name: "default.Table_75f66acf84a545b3ab13614e5f44b929" modify_external_catalogs: true
I20250114 02:54:10.067248 1257 tablet.cc:1721] T 2837990edc4d4bea871a8114533c9352 P 525198593c8e47afaee54e74267b87e3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250114 02:54:10.068714 469 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_63284ca278eb44d8b01f8e7c34a7a6e3" } new_table_name: "default.Table_b146341dc9614d3d92e92e0ecfe60405" modify_external_catalogs: true
I20250114 02:54:10.076300 1207 tablet_bootstrap.cc:654] T 34b7a95953f043a08d1bce25d7af4c00 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:10.081488 1205 tablet_bootstrap.cc:492] T 180a3965833b4ac29467e5207e45d500 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:10.096594 1223 ts_tablet_manager.cc:1918] T 54f9bb2d2e334f53b0c0560f26bbfa3f P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:10.097000 1223 log.cc:1198] T 54f9bb2d2e334f53b0c0560f26bbfa3f P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/54f9bb2d2e334f53b0c0560f26bbfa3f
I20250114 02:54:10.097631 1223 ts_tablet_manager.cc:1939] T 54f9bb2d2e334f53b0c0560f26bbfa3f P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
I20250114 02:54:10.099928 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet 54f9bb2d2e334f53b0c0560f26bbfa3f (table default.Table_c6640103d84c48beb554d7509b246b27 [id=4873b55a952644e3919e7759a9c59b07]) successfully deleted
I20250114 02:54:10.102314 1311 tablet.cc:1721] T 2837990edc4d4bea871a8114533c9352 P 819f9e3412e54d44abb11d8b3ca877c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250114 02:54:10.102576 467 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_075cadc0bfa145718aa9f2f6850f595c" } modify_external_catalogs: true
I20250114 02:54:10.103111 467 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_075cadc0bfa145718aa9f2f6850f595c" } modify_external_catalogs: true
I20250114 02:54:10.108748 1206 ts_tablet_manager.cc:1428] T 34b7a95953f043a08d1bce25d7af4c00 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.109s user 0.004s sys 0.003s
I20250114 02:54:10.111349 465 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet ba074d4ea17747d5aa9287bdaf541681 (table default.Table_017604272128419e9c9d1bba85e595cb [id=253dc2a0f94149a78b0f4ab001fba488]). Expected version 1 got 0
I20250114 02:54:10.111626 1206 tablet_bootstrap.cc:492] T 180a3965833b4ac29467e5207e45d500 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:10.125838 1223 tablet_replica.cc:331] T ee4ed8205f5641bf96b3f617d7f81321 P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:10.126499 1223 raft_consensus.cc:2238] T ee4ed8205f5641bf96b3f617d7f81321 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:10.126888 1223 raft_consensus.cc:2267] T ee4ed8205f5641bf96b3f617d7f81321 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:10.135114 1206 tablet_bootstrap.cc:654] T 180a3965833b4ac29467e5207e45d500 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:10.136284 1223 ts_tablet_manager.cc:1905] T ee4ed8205f5641bf96b3f617d7f81321 P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:10.143023 1205 tablet_bootstrap.cc:654] T 180a3965833b4ac29467e5207e45d500 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
W20250114 02:54:10.143751 462 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b63dd90f60a04465b948171e74888e79 has no consensus state
I20250114 02:54:10.144958 1207 tablet_bootstrap.cc:492] T 34b7a95953f043a08d1bce25d7af4c00 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:10.145357 1207 ts_tablet_manager.cc:1397] T 34b7a95953f043a08d1bce25d7af4c00 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.105s user 0.011s sys 0.005s
I20250114 02:54:10.147955 1207 raft_consensus.cc:357] T 34b7a95953f043a08d1bce25d7af4c00 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:10.148876 1207 raft_consensus.cc:383] T 34b7a95953f043a08d1bce25d7af4c00 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:10.149113 1207 raft_consensus.cc:738] T 34b7a95953f043a08d1bce25d7af4c00 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:10.149691 1207 consensus_queue.cc:260] T 34b7a95953f043a08d1bce25d7af4c00 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:10.152149 1207 ts_tablet_manager.cc:1428] T 34b7a95953f043a08d1bce25d7af4c00 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250114 02:54:10.152944 1207 tablet_bootstrap.cc:492] T a456420f78714cf895528e19dfe0ee8d P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:10.163733 1207 tablet_bootstrap.cc:654] T a456420f78714cf895528e19dfe0ee8d P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:10.174237 1205 tablet_bootstrap.cc:492] T 180a3965833b4ac29467e5207e45d500 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:10.184532 1294 raft_consensus.cc:491] T ba074d4ea17747d5aa9287bdaf541681 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:10.184978 1294 raft_consensus.cc:513] T ba074d4ea17747d5aa9287bdaf541681 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:10.186784 1294 leader_election.cc:290] T ba074d4ea17747d5aa9287bdaf541681 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:10.187548 462 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_a1731c2cc5964c3aaddb43ec22096aad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:10.187873 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba074d4ea17747d5aa9287bdaf541681" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:10.187500 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba074d4ea17747d5aa9287bdaf541681" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:10.188537 962 raft_consensus.cc:2463] T ba074d4ea17747d5aa9287bdaf541681 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 0.
I20250114 02:54:10.189703 751 leader_election.cc:304] T ba074d4ea17747d5aa9287bdaf541681 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
W20250114 02:54:10.190184 753 leader_election.cc:343] T ba074d4ea17747d5aa9287bdaf541681 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
W20250114 02:54:10.190032 462 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a1731c2cc5964c3aaddb43ec22096aad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:10.193652 471 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_47464943699c413b9c925aa383bfe64a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:10.195289 471 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_47464943699c413b9c925aa383bfe64a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:10.197563 1294 raft_consensus.cc:2798] T ba074d4ea17747d5aa9287bdaf541681 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:10.197878 1294 raft_consensus.cc:491] T ba074d4ea17747d5aa9287bdaf541681 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:10.198153 1294 raft_consensus.cc:3054] T ba074d4ea17747d5aa9287bdaf541681 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:10.201313 1223 ts_tablet_manager.cc:1918] T ee4ed8205f5641bf96b3f617d7f81321 P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:10.201653 1223 log.cc:1198] T ee4ed8205f5641bf96b3f617d7f81321 P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/ee4ed8205f5641bf96b3f617d7f81321
I20250114 02:54:10.202241 1223 ts_tablet_manager.cc:1939] T ee4ed8205f5641bf96b3f617d7f81321 P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
I20250114 02:54:10.204324 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet ee4ed8205f5641bf96b3f617d7f81321 (table default.Table_bf49e6df9e43470da41cf3037638033a [id=e8c1f073ed504963947cab370eb9cf1a]) successfully deleted
I20250114 02:54:10.210968 1205 ts_tablet_manager.cc:1397] T 180a3965833b4ac29467e5207e45d500 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.132s user 0.006s sys 0.005s
I20250114 02:54:10.210716 1294 raft_consensus.cc:513] T ba074d4ea17747d5aa9287bdaf541681 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:10.216727 1294 leader_election.cc:290] T ba074d4ea17747d5aa9287bdaf541681 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Requested vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:10.217512 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba074d4ea17747d5aa9287bdaf541681" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:10.218248 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba074d4ea17747d5aa9287bdaf541681" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
I20250114 02:54:10.218776 962 raft_consensus.cc:3054] T ba074d4ea17747d5aa9287bdaf541681 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:10.221410 1207 tablet_bootstrap.cc:492] T a456420f78714cf895528e19dfe0ee8d P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:10.221781 1207 ts_tablet_manager.cc:1397] T a456420f78714cf895528e19dfe0ee8d P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.069s user 0.012s sys 0.000s
I20250114 02:54:10.224920 962 raft_consensus.cc:2463] T ba074d4ea17747d5aa9287bdaf541681 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 1.
I20250114 02:54:10.225596 1207 raft_consensus.cc:357] T a456420f78714cf895528e19dfe0ee8d P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:10.226202 1207 raft_consensus.cc:383] T a456420f78714cf895528e19dfe0ee8d P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:10.226277 1225 ts_tablet_manager.cc:1918] T 54f9bb2d2e334f53b0c0560f26bbfa3f P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:10.226444 1207 raft_consensus.cc:738] T a456420f78714cf895528e19dfe0ee8d P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:10.226771 1225 log.cc:1198] T 54f9bb2d2e334f53b0c0560f26bbfa3f P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/54f9bb2d2e334f53b0c0560f26bbfa3f
I20250114 02:54:10.227443 1225 ts_tablet_manager.cc:1939] T 54f9bb2d2e334f53b0c0560f26bbfa3f P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:10.226996 1207 consensus_queue.cc:260] T a456420f78714cf895528e19dfe0ee8d P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:10.229790 1207 ts_tablet_manager.cc:1428] T a456420f78714cf895528e19dfe0ee8d P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250114 02:54:10.230660 1205 raft_consensus.cc:357] T 180a3965833b4ac29467e5207e45d500 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:10.231696 1205 raft_consensus.cc:383] T 180a3965833b4ac29467e5207e45d500 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:10.232151 1205 raft_consensus.cc:738] T 180a3965833b4ac29467e5207e45d500 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:10.233184 1207 tablet_bootstrap.cc:492] T ec3285525faf482d8739f1bedd653503 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:10.235457 468 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_93b9c6f870f14f6abe67530f6e6d29e8" } modify_external_catalogs: true
I20250114 02:54:10.236014 468 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_93b9c6f870f14f6abe67530f6e6d29e8" } modify_external_catalogs: true
I20250114 02:54:10.244443 1207 tablet_bootstrap.cc:654] T ec3285525faf482d8739f1bedd653503 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:10.254268 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet 54f9bb2d2e334f53b0c0560f26bbfa3f (table default.Table_c6640103d84c48beb554d7509b246b27 [id=4873b55a952644e3919e7759a9c59b07]) successfully deleted
I20250114 02:54:10.233151 1205 consensus_queue.cc:260] T 180a3965833b4ac29467e5207e45d500 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
W20250114 02:54:10.261579 456 catalog_manager.cc:4852] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet 96420dece19549b3876bd795cbdc0f0a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 96420dece19549b3876bd795cbdc0f0a already in progress: creating tablet
I20250114 02:54:10.261797 470 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_f914e27e6a2e4144acdbda801011283b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:10.263358 470 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f914e27e6a2e4144acdbda801011283b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:10.264381 1205 ts_tablet_manager.cc:1428] T 180a3965833b4ac29467e5207e45d500 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.053s user 0.006s sys 0.000s
I20250114 02:54:10.271122 1205 tablet_bootstrap.cc:492] T 34b7a95953f043a08d1bce25d7af4c00 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:10.287577 467 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet a4dd92b0133e413d8b895a0f33e2faf1
I20250114 02:54:10.293556 1084 tablet_service.cc:1514] Processing DeleteTablet for tablet a4dd92b0133e413d8b895a0f33e2faf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:10 UTC) from {username='slave'} at 127.0.0.1:53876
W20250114 02:54:10.295017 458 catalog_manager.cc:4852] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet a4dd92b0133e413d8b895a0f33e2faf1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a4dd92b0133e413d8b895a0f33e2faf1 already in progress: creating tablet
I20250114 02:54:10.297852 937 tablet_service.cc:1514] Processing DeleteTablet for tablet a4dd92b0133e413d8b895a0f33e2faf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:10 UTC) from {username='slave'} at 127.0.0.1:55068
W20250114 02:54:10.299850 456 catalog_manager.cc:4852] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet a4dd92b0133e413d8b895a0f33e2faf1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a4dd92b0133e413d8b895a0f33e2faf1 already in progress: creating tablet
I20250114 02:54:10.324133 466 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 34b7a95953f043a08d1bce25d7af4c00 (table default.Table_c99fdc20f32d41baadd11eb425b279ac [id=c14ed2074c144373a0314f99813e8da3]). Expected version 1 got 0
I20250114 02:54:10.296042 1205 tablet_bootstrap.cc:654] T 34b7a95953f043a08d1bce25d7af4c00 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
W20250114 02:54:10.218586 753 leader_election.cc:343] T ba074d4ea17747d5aa9287bdaf541681 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:10.260777 1206 tablet_bootstrap.cc:492] T 180a3965833b4ac29467e5207e45d500 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:10.327535 1206 ts_tablet_manager.cc:1397] T 180a3965833b4ac29467e5207e45d500 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.216s user 0.012s sys 0.000s
W20250114 02:54:10.327440 753 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.67:45749, user_credentials={real_user=slave}} blocked reactor thread for 108857us
I20250114 02:54:10.298101 1319 raft_consensus.cc:491] T 34b7a95953f043a08d1bce25d7af4c00 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:10.328022 465 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet ba074d4ea17747d5aa9287bdaf541681 (table default.Table_017604272128419e9c9d1bba85e595cb [id=253dc2a0f94149a78b0f4ab001fba488]). Expected version 1 got 0
I20250114 02:54:10.328171 1319 raft_consensus.cc:513] T 34b7a95953f043a08d1bce25d7af4c00 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:10.299309 796 tablet_service.cc:1514] Processing DeleteTablet for tablet a4dd92b0133e413d8b895a0f33e2faf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:10 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:10.330225 1319 leader_election.cc:290] T 34b7a95953f043a08d1bce25d7af4c00 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:10.330753 464 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: default.Table_ba8866c1c90041268e09ff59a46503f1 [id=0b1b2c3855f54d5c97bd2866dfbd8a5d] alter complete (version 3)
I20250114 02:54:10.330704 1206 raft_consensus.cc:357] T 180a3965833b4ac29467e5207e45d500 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:10.331327 1206 raft_consensus.cc:383] T 180a3965833b4ac29467e5207e45d500 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:10.331537 1206 raft_consensus.cc:738] T 180a3965833b4ac29467e5207e45d500 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:10.333460 751 leader_election.cc:304] T ba074d4ea17747d5aa9287bdaf541681 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters: 90d6c2c58bb54e0eafb365625ac33871
W20250114 02:54:10.334028 751 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.66:44037, user_credentials={real_user=slave}} blocked reactor thread for 108183us
I20250114 02:54:10.336897 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34b7a95953f043a08d1bce25d7af4c00" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:10.336944 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34b7a95953f043a08d1bce25d7af4c00" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:10.337515 1106 raft_consensus.cc:2463] T 34b7a95953f043a08d1bce25d7af4c00 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 0.
I20250114 02:54:10.339697 753 leader_election.cc:304] T 34b7a95953f043a08d1bce25d7af4c00 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:10.342954 1319 raft_consensus.cc:2798] T ba074d4ea17747d5aa9287bdaf541681 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:10.344717 464 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet 34b7a95953f043a08d1bce25d7af4c00 (table default.Table_c99fdc20f32d41baadd11eb425b279ac [id=c14ed2074c144373a0314f99813e8da3]). Expected version 1 got 0
I20250114 02:54:10.344538 1235 consensus_queue.cc:1035] T 2a88342b30704cbda0f11f9ddc312fab P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250114 02:54:10.350045 458 catalog_manager.cc:4852] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet a4dd92b0133e413d8b895a0f33e2faf1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a4dd92b0133e413d8b895a0f33e2faf1 already in progress: creating tablet
I20250114 02:54:10.350980 466 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_75f66acf84a545b3ab13614e5f44b929" } new_table_name: "default.Table_9f87e3d3b1234f389cc6b4f39cdf306e" modify_external_catalogs: true
I20250114 02:54:10.354413 1294 raft_consensus.cc:2798] T 34b7a95953f043a08d1bce25d7af4c00 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:10.354557 937 tablet_service.cc:1467] Processing CreateTablet for tablet b4740f6b3a6741368ba68907ad065279 (DEFAULT_TABLE table=default.Table_8a880e0ae5ff4b5d8de8f40f15dd6e02 [id=6baaeac9e83d4e3092485fca13e07303]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:10.354801 1294 raft_consensus.cc:491] T 34b7a95953f043a08d1bce25d7af4c00 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:10.355149 1294 raft_consensus.cc:3054] T 34b7a95953f043a08d1bce25d7af4c00 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:10.355551 937 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4740f6b3a6741368ba68907ad065279. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:10.358099 1084 tablet_service.cc:1467] Processing CreateTablet for tablet b4740f6b3a6741368ba68907ad065279 (DEFAULT_TABLE table=default.Table_8a880e0ae5ff4b5d8de8f40f15dd6e02 [id=6baaeac9e83d4e3092485fca13e07303]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:10.359406 1084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4740f6b3a6741368ba68907ad065279. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:10.363770 796 tablet_service.cc:1467] Processing CreateTablet for tablet b4740f6b3a6741368ba68907ad065279 (DEFAULT_TABLE table=default.Table_8a880e0ae5ff4b5d8de8f40f15dd6e02 [id=6baaeac9e83d4e3092485fca13e07303]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:10.365033 796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4740f6b3a6741368ba68907ad065279. 1 dirs total, 0 dirs full, 0 dirs failed
W20250114 02:54:10.368674 751 leader_election.cc:343] T 34b7a95953f043a08d1bce25d7af4c00 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
W20250114 02:54:10.373777 455 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet b63dd90f60a04465b948171e74888e79 (table default.Table_75f66acf84a545b3ab13614e5f44b929 [id=d8c9691818d042d7b49475fa009c4ca1], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet b63dd90f60a04465b948171e74888e79 has no consensus state
I20250114 02:54:10.372620 1206 consensus_queue.cc:260] T 180a3965833b4ac29467e5207e45d500 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:10.377213 1205 tablet_bootstrap.cc:492] T 34b7a95953f043a08d1bce25d7af4c00 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:10.377586 1205 ts_tablet_manager.cc:1397] T 34b7a95953f043a08d1bce25d7af4c00 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.107s user 0.017s sys 0.006s
I20250114 02:54:10.379102 468 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet 2a88342b30704cbda0f11f9ddc312fab
I20250114 02:54:10.380609 1257 tablet.cc:1721] T a1cf92e5275b454ca893b12f2d90d25b P 525198593c8e47afaee54e74267b87e3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250114 02:54:10.385205 1205 raft_consensus.cc:357] T 34b7a95953f043a08d1bce25d7af4c00 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:10.385847 1205 raft_consensus.cc:383] T 34b7a95953f043a08d1bce25d7af4c00 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:10.386087 1205 raft_consensus.cc:738] T 34b7a95953f043a08d1bce25d7af4c00 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:10.386675 1205 consensus_queue.cc:260] T 34b7a95953f043a08d1bce25d7af4c00 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:10.387306 938 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a88342b30704cbda0f11f9ddc312fab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:10 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:10.388027 1225 tablet_replica.cc:331] T 2a88342b30704cbda0f11f9ddc312fab P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:10.388667 1225 raft_consensus.cc:2238] T 2a88342b30704cbda0f11f9ddc312fab P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Raft consensus shutting down.
I20250114 02:54:10.393493 793 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a88342b30704cbda0f11f9ddc312fab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:10 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:10.393952 1083 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a88342b30704cbda0f11f9ddc312fab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:10 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:10.394207 1222 tablet_replica.cc:331] T 2a88342b30704cbda0f11f9ddc312fab P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:10.394819 1222 raft_consensus.cc:2238] T 2a88342b30704cbda0f11f9ddc312fab P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:10.395254 1222 raft_consensus.cc:2267] T 2a88342b30704cbda0f11f9ddc312fab P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:10.396850 1222 ts_tablet_manager.cc:1905] T 2a88342b30704cbda0f11f9ddc312fab P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:10.400497 1223 tablet_replica.cc:331] T 2a88342b30704cbda0f11f9ddc312fab P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:10.401198 1223 raft_consensus.cc:2238] T 2a88342b30704cbda0f11f9ddc312fab P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:10.401607 1223 raft_consensus.cc:2267] T 2a88342b30704cbda0f11f9ddc312fab P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:10.401731 1206 ts_tablet_manager.cc:1428] T 180a3965833b4ac29467e5207e45d500 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.074s user 0.004s sys 0.003s
I20250114 02:54:10.403193 1223 ts_tablet_manager.cc:1905] T 2a88342b30704cbda0f11f9ddc312fab P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:10.404445 1259 tablet.cc:1721] T a1cf92e5275b454ca893b12f2d90d25b P 90d6c2c58bb54e0eafb365625ac33871: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250114 02:54:10.405606 1206 tablet_bootstrap.cc:492] T ec3285525faf482d8739f1bedd653503 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:10.407387 1083 tablet_service.cc:1467] Processing CreateTablet for tablet 0752ab4b7031402db4e6be8b61cf364f (DEFAULT_TABLE table=default.Table_e691c983a1bc4ce3a3d4ac683c048218 [id=0543f6180069494c8055f8931b9ac3c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:10.408464 1083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0752ab4b7031402db4e6be8b61cf364f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:10.418138 468 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_6c885213c7cb45e2b12955642171daff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:10.427307 468 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6c885213c7cb45e2b12955642171daff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250114 02:54:10.442484 466 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b63dd90f60a04465b948171e74888e79 has no consensus state
I20250114 02:54:10.444983 1206 tablet_bootstrap.cc:654] T ec3285525faf482d8739f1bedd653503 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:10.447855 793 tablet_service.cc:1467] Processing CreateTablet for tablet 0752ab4b7031402db4e6be8b61cf364f (DEFAULT_TABLE table=default.Table_e691c983a1bc4ce3a3d4ac683c048218 [id=0543f6180069494c8055f8931b9ac3c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:10.448920 938 tablet_service.cc:1467] Processing CreateTablet for tablet 0752ab4b7031402db4e6be8b61cf364f (DEFAULT_TABLE table=default.Table_e691c983a1bc4ce3a3d4ac683c048218 [id=0543f6180069494c8055f8931b9ac3c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:10.449218 793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0752ab4b7031402db4e6be8b61cf364f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:10.450251 938 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0752ab4b7031402db4e6be8b61cf364f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:10.455416 1085 tablet_service.cc:1467] Processing CreateTablet for tablet 61b4b6d4ac364a14a34733b4dd715bb3 (DEFAULT_TABLE table=default.Table_2842b4625fde43f4b83c5cf150c6ae93 [id=e279c31bd63b40b9ade03a3b928bf153]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:10.456449 1085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61b4b6d4ac364a14a34733b4dd715bb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:10.470294 466 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_245ce3f7a09e489d92122a15f95bacf7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:10.476884 1205 ts_tablet_manager.cc:1428] T 34b7a95953f043a08d1bce25d7af4c00 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.099s user 0.000s sys 0.007s
I20250114 02:54:10.477734 1205 tablet_bootstrap.cc:492] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
W20250114 02:54:10.477820 898 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.67:45749, user_credentials={real_user=slave}} blocked reactor thread for 81854.2us
I20250114 02:54:10.478572 1225 pending_rounds.cc:62] T 2a88342b30704cbda0f11f9ddc312fab P 525198593c8e47afaee54e74267b87e3: Trying to abort 1 pending ops.
I20250114 02:54:10.478819 1225 pending_rounds.cc:69] T 2a88342b30704cbda0f11f9ddc312fab P 525198593c8e47afaee54e74267b87e3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114028029617192960 op_type: NO_OP noop_request { }
I20250114 02:54:10.479193 1225 raft_consensus.cc:2883] T 2a88342b30704cbda0f11f9ddc312fab P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250114 02:54:10.479477 1225 raft_consensus.cc:2267] T 2a88342b30704cbda0f11f9ddc312fab P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250114 02:54:10.480276 466 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_245ce3f7a09e489d92122a15f95bacf7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:10.480721 1249 raft_consensus.cc:491] T 34b7a95953f043a08d1bce25d7af4c00 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:10.481706 1225 ts_tablet_manager.cc:1905] T 2a88342b30704cbda0f11f9ddc312fab P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:10.481205 1249 raft_consensus.cc:513] T 34b7a95953f043a08d1bce25d7af4c00 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:10.485759 1249 leader_election.cc:290] T 34b7a95953f043a08d1bce25d7af4c00 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:10.486001 1222 ts_tablet_manager.cc:1918] T 2a88342b30704cbda0f11f9ddc312fab P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:10.486338 1222 log.cc:1198] T 2a88342b30704cbda0f11f9ddc312fab P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/2a88342b30704cbda0f11f9ddc312fab
I20250114 02:54:10.486934 1222 ts_tablet_manager.cc:1939] T 2a88342b30704cbda0f11f9ddc312fab P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:10.486757 467 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_c6ab81139a97463d81e5a7f6334a4006"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:10.487577 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34b7a95953f043a08d1bce25d7af4c00" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:10.488164 1106 raft_consensus.cc:2463] T 34b7a95953f043a08d1bce25d7af4c00 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
W20250114 02:54:10.488404 467 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c6ab81139a97463d81e5a7f6334a4006 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:10.490398 792 tablet_service.cc:1467] Processing CreateTablet for tablet 61b4b6d4ac364a14a34733b4dd715bb3 (DEFAULT_TABLE table=default.Table_2842b4625fde43f4b83c5cf150c6ae93 [id=e279c31bd63b40b9ade03a3b928bf153]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:10.491413 792 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61b4b6d4ac364a14a34733b4dd715bb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:10.494570 470 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_bcf55148b6bc413e8d4e8e31996ff32b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:10.495271 898 leader_election.cc:304] T 34b7a95953f043a08d1bce25d7af4c00 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters:
W20250114 02:54:10.496295 470 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bcf55148b6bc413e8d4e8e31996ff32b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:10.497179 1249 raft_consensus.cc:2798] T 34b7a95953f043a08d1bce25d7af4c00 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:10.497473 1249 raft_consensus.cc:491] T 34b7a95953f043a08d1bce25d7af4c00 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:10.497838 1249 raft_consensus.cc:3054] T 34b7a95953f043a08d1bce25d7af4c00 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:10.499351 1225 ts_tablet_manager.cc:1918] T 2a88342b30704cbda0f11f9ddc312fab P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:10.499675 1225 log.cc:1198] T 2a88342b30704cbda0f11f9ddc312fab P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/2a88342b30704cbda0f11f9ddc312fab
I20250114 02:54:10.495960 1319 raft_consensus.cc:695] T ba074d4ea17747d5aa9287bdaf541681 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 LEADER]: Becoming Leader. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Running, Role: LEADER
I20250114 02:54:10.502705 1225 ts_tablet_manager.cc:1939] T 2a88342b30704cbda0f11f9ddc312fab P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:10.504673 464 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_97368d1825cf41e981b17a5e7c8d2148" } modify_external_catalogs: true
I20250114 02:54:10.505105 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet 2a88342b30704cbda0f11f9ddc312fab (table default.Table_93b9c6f870f14f6abe67530f6e6d29e8 [id=b1e4e75afa524c68b5976a885116ef1c]) successfully deleted
I20250114 02:54:10.505331 464 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_97368d1825cf41e981b17a5e7c8d2148" } modify_external_catalogs: true
I20250114 02:54:10.508041 937 tablet_service.cc:1467] Processing CreateTablet for tablet 61b4b6d4ac364a14a34733b4dd715bb3 (DEFAULT_TABLE table=default.Table_2842b4625fde43f4b83c5cf150c6ae93 [id=e279c31bd63b40b9ade03a3b928bf153]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:10.510586 937 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61b4b6d4ac364a14a34733b4dd715bb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:10.515193 1223 ts_tablet_manager.cc:1918] T 2a88342b30704cbda0f11f9ddc312fab P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:10.515570 1223 log.cc:1198] T 2a88342b30704cbda0f11f9ddc312fab P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/2a88342b30704cbda0f11f9ddc312fab
I20250114 02:54:10.518286 1207 tablet_bootstrap.cc:492] T ec3285525faf482d8739f1bedd653503 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:10.518715 1207 ts_tablet_manager.cc:1397] T ec3285525faf482d8739f1bedd653503 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.286s user 0.036s sys 0.046s
I20250114 02:54:10.520314 1249 raft_consensus.cc:513] T 34b7a95953f043a08d1bce25d7af4c00 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:10.521257 465 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: default.Table_b146341dc9614d3d92e92e0ecfe60405 [id=9ec503e449fb4a2a96bbb77237c2fb4f] alter complete (version 4)
I20250114 02:54:10.521082 1207 raft_consensus.cc:357] T ec3285525faf482d8739f1bedd653503 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:10.521723 1207 raft_consensus.cc:383] T ec3285525faf482d8739f1bedd653503 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:10.521978 1207 raft_consensus.cc:738] T ec3285525faf482d8739f1bedd653503 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:10.524612 1207 consensus_queue.cc:260] T ec3285525faf482d8739f1bedd653503 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:10.526826 1205 tablet_bootstrap.cc:654] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:10.530256 469 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 34b7a95953f043a08d1bce25d7af4c00 (table default.Table_c99fdc20f32d41baadd11eb425b279ac [id=c14ed2074c144373a0314f99813e8da3]). Expected version 1 got 0
I20250114 02:54:10.534054 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34b7a95953f043a08d1bce25d7af4c00" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:10.534642 1106 raft_consensus.cc:3054] T 34b7a95953f043a08d1bce25d7af4c00 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:10.535079 1207 ts_tablet_manager.cc:1428] T ec3285525faf482d8739f1bedd653503 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
I20250114 02:54:10.535949 1207 tablet_bootstrap.cc:492] T ba074d4ea17747d5aa9287bdaf541681 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:10.540688 462 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_2842b4625fde43f4b83c5cf150c6ae93" } new_table_name: "default.Table_893071c3e2ec472a81de4617fc929071" modify_external_catalogs: true
I20250114 02:54:10.544586 1235 raft_consensus.cc:491] T 180a3965833b4ac29467e5207e45d500 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:10.545045 1235 raft_consensus.cc:513] T 180a3965833b4ac29467e5207e45d500 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:10.546662 1235 leader_election.cc:290] T 180a3965833b4ac29467e5207e45d500 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:10.548561 1207 tablet_bootstrap.cc:654] T ba074d4ea17747d5aa9287bdaf541681 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:10.548333 1104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "180a3965833b4ac29467e5207e45d500" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:10.552942 1249 leader_election.cc:290] T 34b7a95953f043a08d1bce25d7af4c00 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Requested vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:10.553789 1223 ts_tablet_manager.cc:1939] T 2a88342b30704cbda0f11f9ddc312fab P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
W20250114 02:54:10.555503 898 leader_election.cc:343] T 180a3965833b4ac29467e5207e45d500 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:10.501852 1319 consensus_queue.cc:237] T ba074d4ea17747d5aa9287bdaf541681 P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
W20250114 02:54:10.558734 753 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.67:45749, user_credentials={real_user=slave}} blocked reactor thread for 219113us
I20250114 02:54:10.572652 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet 2a88342b30704cbda0f11f9ddc312fab (table default.Table_93b9c6f870f14f6abe67530f6e6d29e8 [id=b1e4e75afa524c68b5976a885116ef1c]) successfully deleted
I20250114 02:54:10.575775 1106 raft_consensus.cc:2463] T 34b7a95953f043a08d1bce25d7af4c00 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:10.577117 898 leader_election.cc:304] T 34b7a95953f043a08d1bce25d7af4c00 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:10.578060 1249 raft_consensus.cc:2798] T 34b7a95953f043a08d1bce25d7af4c00 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:10.581694 1249 raft_consensus.cc:695] T 34b7a95953f043a08d1bce25d7af4c00 P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Becoming Leader. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Running, Role: LEADER
I20250114 02:54:10.584192 817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34b7a95953f043a08d1bce25d7af4c00" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:10.586843 815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34b7a95953f043a08d1bce25d7af4c00" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:10.588524 1207 tablet_bootstrap.cc:492] T ba074d4ea17747d5aa9287bdaf541681 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:10.588918 1207 ts_tablet_manager.cc:1397] T ba074d4ea17747d5aa9287bdaf541681 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.053s user 0.010s sys 0.005s
I20250114 02:54:10.591156 1207 raft_consensus.cc:357] T ba074d4ea17747d5aa9287bdaf541681 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:10.589684 1249 consensus_queue.cc:237] T 34b7a95953f043a08d1bce25d7af4c00 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:10.594630 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet 2a88342b30704cbda0f11f9ddc312fab (table default.Table_93b9c6f870f14f6abe67530f6e6d29e8 [id=b1e4e75afa524c68b5976a885116ef1c]) successfully deleted
I20250114 02:54:10.594390 816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "180a3965833b4ac29467e5207e45d500" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:10.596025 1207 raft_consensus.cc:383] T ba074d4ea17747d5aa9287bdaf541681 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:10.596324 1207 raft_consensus.cc:738] T ba074d4ea17747d5aa9287bdaf541681 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:10.596880 1207 consensus_queue.cc:260] T ba074d4ea17747d5aa9287bdaf541681 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:10.600392 1207 ts_tablet_manager.cc:1428] T ba074d4ea17747d5aa9287bdaf541681 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20250114 02:54:10.601680 467 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_24dc0a51cadc4387918d43a65986ebf4" } modify_external_catalogs: true
I20250114 02:54:10.602190 467 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_24dc0a51cadc4387918d43a65986ebf4" } modify_external_catalogs: true
I20250114 02:54:10.604902 466 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet ba074d4ea17747d5aa9287bdaf541681 (table default.Table_017604272128419e9c9d1bba85e595cb [id=253dc2a0f94149a78b0f4ab001fba488]). Expected version 1 got 0
I20250114 02:54:10.606226 816 raft_consensus.cc:2463] T 180a3965833b4ac29467e5207e45d500 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:10.612587 898 leader_election.cc:304] T 180a3965833b4ac29467e5207e45d500 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters: 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:10.613601 1235 raft_consensus.cc:2798] T 180a3965833b4ac29467e5207e45d500 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:10.613924 1235 raft_consensus.cc:491] T 180a3965833b4ac29467e5207e45d500 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:10.614210 1235 raft_consensus.cc:3054] T 180a3965833b4ac29467e5207e45d500 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:10.614604 1207 tablet_bootstrap.cc:492] T 96420dece19549b3876bd795cbdc0f0a P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:10.587226 1205 tablet_bootstrap.cc:492] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:10.617974 1205 ts_tablet_manager.cc:1397] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.140s user 0.008s sys 0.004s
I20250114 02:54:10.620086 1205 raft_consensus.cc:357] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:10.620820 1205 raft_consensus.cc:383] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:10.621068 1205 raft_consensus.cc:738] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:10.621471 464 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet 180a3965833b4ac29467e5207e45d500
I20250114 02:54:10.621160 1235 raft_consensus.cc:513] T 180a3965833b4ac29467e5207e45d500 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:10.623710 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "180a3965833b4ac29467e5207e45d500" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:10.624620 1207 tablet_bootstrap.cc:654] T 96420dece19549b3876bd795cbdc0f0a P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
W20250114 02:54:10.625118 898 leader_election.cc:343] T 180a3965833b4ac29467e5207e45d500 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:10.627478 1205 consensus_queue.cc:260] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:10.633854 1205 ts_tablet_manager.cc:1428] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.016s user 0.007s sys 0.000s
I20250114 02:54:10.635182 1205 tablet_bootstrap.cc:492] T a4dd92b0133e413d8b895a0f33e2faf1 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:10.636734 1083 tablet_service.cc:1514] Processing DeleteTablet for tablet 180a3965833b4ac29467e5207e45d500 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:10 UTC) from {username='slave'} at 127.0.0.1:53876
W20250114 02:54:10.642053 458 catalog_manager.cc:4852] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 180a3965833b4ac29467e5207e45d500 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 180a3965833b4ac29467e5207e45d500 already in progress: creating tablet
I20250114 02:54:10.643465 1294 raft_consensus.cc:513] T 34b7a95953f043a08d1bce25d7af4c00 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:10.644531 817 raft_consensus.cc:2388] T 34b7a95953f043a08d1bce25d7af4c00 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 525198593c8e47afaee54e74267b87e3 in current term 1: Already voted for candidate 819f9e3412e54d44abb11d8b3ca877c2 in this term.
I20250114 02:54:10.645196 1294 leader_election.cc:290] T 34b7a95953f043a08d1bce25d7af4c00 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Requested vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:10.645749 470 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_4720eed4e94c42f589ad0117109d5d77" } new_table_name: "default.Table_35a78676616e4b20906b262c9d0c0d9e" modify_external_catalogs: true
I20250114 02:54:10.650481 1205 tablet_bootstrap.cc:654] T a4dd92b0133e413d8b895a0f33e2faf1 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:10.650728 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34b7a95953f043a08d1bce25d7af4c00" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
I20250114 02:54:10.652480 817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "180a3965833b4ac29467e5207e45d500" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:10.653097 817 raft_consensus.cc:3054] T 180a3965833b4ac29467e5207e45d500 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:10.656560 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34b7a95953f043a08d1bce25d7af4c00" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:10.657346 1106 raft_consensus.cc:2388] T 34b7a95953f043a08d1bce25d7af4c00 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 819f9e3412e54d44abb11d8b3ca877c2 in current term 1: Already voted for candidate 525198593c8e47afaee54e74267b87e3 in this term.
W20250114 02:54:10.658190 462 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 61b4b6d4ac364a14a34733b4dd715bb3 consensus state has no leader
I20250114 02:54:10.658417 753 leader_election.cc:304] T 34b7a95953f043a08d1bce25d7af4c00 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:10.659468 1294 raft_consensus.cc:2743] T 34b7a95953f043a08d1bce25d7af4c00 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250114 02:54:10.660094 1235 leader_election.cc:290] T 180a3965833b4ac29467e5207e45d500 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Requested vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:10.665359 467 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet c69d5eef3f3f49699e76dcbe1be7d6f6
I20250114 02:54:10.669243 1083 tablet_service.cc:1514] Processing DeleteTablet for tablet c69d5eef3f3f49699e76dcbe1be7d6f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:10 UTC) from {username='slave'} at 127.0.0.1:53876
W20250114 02:54:10.670656 458 catalog_manager.cc:4852] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet c69d5eef3f3f49699e76dcbe1be7d6f6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c69d5eef3f3f49699e76dcbe1be7d6f6 already in progress: creating tablet
I20250114 02:54:10.679876 938 tablet_service.cc:1514] Processing DeleteTablet for tablet 180a3965833b4ac29467e5207e45d500 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:10 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:10.680428 937 tablet_service.cc:1514] Processing DeleteTablet for tablet c69d5eef3f3f49699e76dcbe1be7d6f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:10 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:10.664191 817 raft_consensus.cc:2463] T 180a3965833b4ac29467e5207e45d500 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:10.669917 1311 tablet.cc:1721] T a1cf92e5275b454ca893b12f2d90d25b P 819f9e3412e54d44abb11d8b3ca877c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250114 02:54:10.678987 796 tablet_service.cc:1514] Processing DeleteTablet for tablet c69d5eef3f3f49699e76dcbe1be7d6f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:10 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:10.666504 1206 tablet_bootstrap.cc:492] T ec3285525faf482d8739f1bedd653503 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
W20250114 02:54:10.684751 470 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a3886c5f976445cebc95ef1839a74aa9 consensus state has no leader
I20250114 02:54:10.684911 1206 ts_tablet_manager.cc:1397] T ec3285525faf482d8739f1bedd653503 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.279s user 0.004s sys 0.016s
I20250114 02:54:10.685433 1225 tablet_replica.cc:331] T 180a3965833b4ac29467e5207e45d500 P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
W20250114 02:54:10.685951 458 catalog_manager.cc:4852] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet c69d5eef3f3f49699e76dcbe1be7d6f6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c69d5eef3f3f49699e76dcbe1be7d6f6 already in progress: creating tablet
I20250114 02:54:10.686148 1225 raft_consensus.cc:2238] T 180a3965833b4ac29467e5207e45d500 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:10.686547 1225 raft_consensus.cc:2267] T 180a3965833b4ac29467e5207e45d500 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:10.688097 1225 ts_tablet_manager.cc:1905] T 180a3965833b4ac29467e5207e45d500 P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:10.690821 1207 tablet_bootstrap.cc:492] T 96420dece19549b3876bd795cbdc0f0a P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:10.691224 1207 ts_tablet_manager.cc:1397] T 96420dece19549b3876bd795cbdc0f0a P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.077s user 0.010s sys 0.000s
I20250114 02:54:10.693593 1207 raft_consensus.cc:357] T 96420dece19549b3876bd795cbdc0f0a P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:10.694226 1207 raft_consensus.cc:383] T 96420dece19549b3876bd795cbdc0f0a P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:10.694456 1207 raft_consensus.cc:738] T 96420dece19549b3876bd795cbdc0f0a P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:10.694990 1207 consensus_queue.cc:260] T 96420dece19549b3876bd795cbdc0f0a P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:10.700044 898 leader_election.cc:304] T 180a3965833b4ac29467e5207e45d500 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters: 90d6c2c58bb54e0eafb365625ac33871
W20250114 02:54:10.700837 898 raft_consensus.cc:2681] T 180a3965833b4ac29467e5207e45d500 P 525198593c8e47afaee54e74267b87e3: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250114 02:54:10.701800 1207 ts_tablet_manager.cc:1428] T 96420dece19549b3876bd795cbdc0f0a P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20250114 02:54:10.703707 1207 tablet_bootstrap.cc:492] T 180a3965833b4ac29467e5207e45d500 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:10.687079 1206 raft_consensus.cc:357] T ec3285525faf482d8739f1bedd653503 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:10.704413 1206 raft_consensus.cc:383] T ec3285525faf482d8739f1bedd653503 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:10.704653 1206 raft_consensus.cc:738] T ec3285525faf482d8739f1bedd653503 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:10.705255 1206 consensus_queue.cc:260] T ec3285525faf482d8739f1bedd653503 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:10.709761 1206 ts_tablet_manager.cc:1428] T ec3285525faf482d8739f1bedd653503 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.025s user 0.003s sys 0.002s
I20250114 02:54:10.713949 792 tablet_service.cc:1514] Processing DeleteTablet for tablet 180a3965833b4ac29467e5207e45d500 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:10 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:10.714601 1222 tablet_replica.cc:331] T 180a3965833b4ac29467e5207e45d500 P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:10.715232 1222 raft_consensus.cc:2238] T 180a3965833b4ac29467e5207e45d500 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:10.715626 1222 raft_consensus.cc:2267] T 180a3965833b4ac29467e5207e45d500 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:10.716096 1106 raft_consensus.cc:3054] T a456420f78714cf895528e19dfe0ee8d P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:10.717584 1222 ts_tablet_manager.cc:1905] T 180a3965833b4ac29467e5207e45d500 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:10.719607 1207 tablet_bootstrap.cc:654] T 180a3965833b4ac29467e5207e45d500 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:10.724814 1294 consensus_queue.cc:1035] T a456420f78714cf895528e19dfe0ee8d P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:54:10.738183 1206 tablet_bootstrap.cc:492] T 96420dece19549b3876bd795cbdc0f0a P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:10.738238 464 catalog_manager.cc:5526] T ba074d4ea17747d5aa9287bdaf541681 P 819f9e3412e54d44abb11d8b3ca877c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65). New cstate: current_term: 1 leader_uuid: "819f9e3412e54d44abb11d8b3ca877c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:10.742461 796 tablet_service.cc:1467] Processing CreateTablet for tablet 276942e5e54246c79f0d4f08297f58ea (DEFAULT_TABLE table=default.Table_47464943699c413b9c925aa383bfe64a [id=c3d72f049d9849ab86e252c47e5d24e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:10.743655 796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 276942e5e54246c79f0d4f08297f58ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:10.745383 1235 raft_consensus.cc:491] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:10.746026 1235 raft_consensus.cc:513] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:10.747826 464 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet ba074d4ea17747d5aa9287bdaf541681 (table default.Table_017604272128419e9c9d1bba85e595cb [id=253dc2a0f94149a78b0f4ab001fba488]). Expected version 1 got 0
I20250114 02:54:10.748647 1235 leader_election.cc:290] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:10.754366 1104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c69d5eef3f3f49699e76dcbe1be7d6f6" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:10.754405 470 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_6a4fe39298514187b65def5841d3a992"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:10.755797 898 leader_election.cc:343] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
W20250114 02:54:10.755975 470 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6a4fe39298514187b65def5841d3a992 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:10.756917 817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c69d5eef3f3f49699e76dcbe1be7d6f6" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
W20250114 02:54:10.758751 898 leader_election.cc:343] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:10.759310 898 leader_election.cc:304] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 525198593c8e47afaee54e74267b87e3; no voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:10.761080 1235 raft_consensus.cc:2743] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:10.762099 1083 tablet_service.cc:1467] Processing CreateTablet for tablet 276942e5e54246c79f0d4f08297f58ea (DEFAULT_TABLE table=default.Table_47464943699c413b9c925aa383bfe64a [id=c3d72f049d9849ab86e252c47e5d24e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:10.763052 1083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 276942e5e54246c79f0d4f08297f58ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:10.769855 466 catalog_manager.cc:5526] T 34b7a95953f043a08d1bce25d7af4c00 P 525198593c8e47afaee54e74267b87e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 525198593c8e47afaee54e74267b87e3 (127.0.107.66). New cstate: current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } }
I20250114 02:54:10.771139 466 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 34b7a95953f043a08d1bce25d7af4c00 (table default.Table_c99fdc20f32d41baadd11eb425b279ac [id=c14ed2074c144373a0314f99813e8da3]). Expected version 1 got 0
I20250114 02:54:10.774173 1309 raft_consensus.cc:491] T ec3285525faf482d8739f1bedd653503 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:10.774576 1309 raft_consensus.cc:513] T ec3285525faf482d8739f1bedd653503 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:10.774749 793 tablet_service.cc:1467] Processing CreateTablet for tablet b3b1ea0a825c4d1f8e3edf24cc74d8f7 (DEFAULT_TABLE table=default.Table_f914e27e6a2e4144acdbda801011283b [id=45d8444e54064c709c13103963c1e9ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:10.775777 793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3b1ea0a825c4d1f8e3edf24cc74d8f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:10.778313 1309 leader_election.cc:290] T ec3285525faf482d8739f1bedd653503 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:10.778966 467 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_1f08339549de49f5b5abc889a248657c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:10.779632 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec3285525faf482d8739f1bedd653503" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:10.780210 962 raft_consensus.cc:2463] T ec3285525faf482d8739f1bedd653503 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 0.
I20250114 02:54:10.780268 817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec3285525faf482d8739f1bedd653503" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
W20250114 02:54:10.780539 467 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1f08339549de49f5b5abc889a248657c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:10.781152 1040 leader_election.cc:304] T ec3285525faf482d8739f1bedd653503 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:10.781777 817 raft_consensus.cc:2463] T ec3285525faf482d8739f1bedd653503 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 0.
I20250114 02:54:10.782451 1309 raft_consensus.cc:2798] T ec3285525faf482d8739f1bedd653503 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:10.782759 1309 raft_consensus.cc:491] T ec3285525faf482d8739f1bedd653503 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:10.783079 1309 raft_consensus.cc:3054] T ec3285525faf482d8739f1bedd653503 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:10.782881 1321 consensus_queue.cc:1035] T a456420f78714cf895528e19dfe0ee8d P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:54:10.794766 1085 tablet_service.cc:1467] Processing CreateTablet for tablet b3b1ea0a825c4d1f8e3edf24cc74d8f7 (DEFAULT_TABLE table=default.Table_f914e27e6a2e4144acdbda801011283b [id=45d8444e54064c709c13103963c1e9ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:10.795773 1085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3b1ea0a825c4d1f8e3edf24cc74d8f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:10.795805 1206 tablet_bootstrap.cc:654] T 96420dece19549b3876bd795cbdc0f0a P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:10.793991 938 tablet_service.cc:1467] Processing CreateTablet for tablet 276942e5e54246c79f0d4f08297f58ea (DEFAULT_TABLE table=default.Table_47464943699c413b9c925aa383bfe64a [id=c3d72f049d9849ab86e252c47e5d24e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:10.797474 462 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_6741923eb4f845ddbe5844a0e21b38bf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:10.798936 462 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6741923eb4f845ddbe5844a0e21b38bf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:10.803041 465 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet ba074d4ea17747d5aa9287bdaf541681 (table default.Table_017604272128419e9c9d1bba85e595cb [id=253dc2a0f94149a78b0f4ab001fba488]). Expected version 1 got 0
I20250114 02:54:10.820864 937 tablet_service.cc:1467] Processing CreateTablet for tablet b3b1ea0a825c4d1f8e3edf24cc74d8f7 (DEFAULT_TABLE table=default.Table_f914e27e6a2e4144acdbda801011283b [id=45d8444e54064c709c13103963c1e9ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:10.821811 937 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3b1ea0a825c4d1f8e3edf24cc74d8f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:10.823952 470 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_6e385f4df79b47cb9f1ada084a60abbf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:10.825565 470 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6e385f4df79b47cb9f1ada084a60abbf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:10.828462 465 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_a1731c2cc5964c3aaddb43ec22096aad" } modify_external_catalogs: true
I20250114 02:54:10.829062 465 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_a1731c2cc5964c3aaddb43ec22096aad" } modify_external_catalogs: true
I20250114 02:54:10.831715 936 tablet_service.cc:1467] Processing CreateTablet for tablet 8b4bc535c42e43889401f14b6d59659e (DEFAULT_TABLE table=default.Table_b5367d1b4b2a4ec5a97350612e373c40 [id=f8e40e0aa44b4157ac3438d11685b2e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:10.832660 936 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b4bc535c42e43889401f14b6d59659e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:10.834434 1081 tablet_service.cc:1467] Processing CreateTablet for tablet 4bb9b429af92446bbe13894d5dfe4c47 (DEFAULT_TABLE table=default.Table_2a1287091fb9466d9e2fae14499a0963 [id=22b7edcf4b7e4c31b3c23dc44ccf028e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:10.835397 1081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bb9b429af92446bbe13894d5dfe4c47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:10.836920 794 tablet_service.cc:1467] Processing CreateTablet for tablet 4bb9b429af92446bbe13894d5dfe4c47 (DEFAULT_TABLE table=default.Table_2a1287091fb9466d9e2fae14499a0963 [id=22b7edcf4b7e4c31b3c23dc44ccf028e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:10.837436 1084 tablet_service.cc:1467] Processing CreateTablet for tablet 8b4bc535c42e43889401f14b6d59659e (DEFAULT_TABLE table=default.Table_b5367d1b4b2a4ec5a97350612e373c40 [id=f8e40e0aa44b4157ac3438d11685b2e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:10.837877 794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bb9b429af92446bbe13894d5dfe4c47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:10.838411 1084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b4bc535c42e43889401f14b6d59659e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:10.841558 1309 raft_consensus.cc:513] T ec3285525faf482d8739f1bedd653503 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:10.844596 817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec3285525faf482d8739f1bedd653503" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:10.845180 817 raft_consensus.cc:3054] T ec3285525faf482d8739f1bedd653503 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:10.840085 938 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 276942e5e54246c79f0d4f08297f58ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:10.865011 469 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_7a318166ed3045c7bbae5b779be062c4" } new_table_name: "default.Table_acca64d3328d47578d6d1ebd594babbd" modify_external_catalogs: true
I20250114 02:54:10.865177 792 tablet_service.cc:1467] Processing CreateTablet for tablet 8b4bc535c42e43889401f14b6d59659e (DEFAULT_TABLE table=default.Table_b5367d1b4b2a4ec5a97350612e373c40 [id=f8e40e0aa44b4157ac3438d11685b2e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:10.866253 792 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b4bc535c42e43889401f14b6d59659e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:10.866456 1206 tablet_bootstrap.cc:492] T 96420dece19549b3876bd795cbdc0f0a P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:10.866834 1206 ts_tablet_manager.cc:1397] T 96420dece19549b3876bd795cbdc0f0a P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.129s user 0.004s sys 0.004s
I20250114 02:54:10.869089 1206 raft_consensus.cc:357] T 96420dece19549b3876bd795cbdc0f0a P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:10.869659 1206 raft_consensus.cc:383] T 96420dece19549b3876bd795cbdc0f0a P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:10.869899 1206 raft_consensus.cc:738] T 96420dece19549b3876bd795cbdc0f0a P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:10.870157 468 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_c99fdc20f32d41baadd11eb425b279ac" } modify_external_catalogs: true
I20250114 02:54:10.870675 468 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_c99fdc20f32d41baadd11eb425b279ac" } modify_external_catalogs: true
I20250114 02:54:10.870451 1206 consensus_queue.cc:260] T 96420dece19549b3876bd795cbdc0f0a P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:10.873051 817 raft_consensus.cc:2463] T ec3285525faf482d8739f1bedd653503 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1.
I20250114 02:54:10.874033 1042 leader_election.cc:304] T ec3285525faf482d8739f1bedd653503 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:10.883163 1338 raft_consensus.cc:491] T 96420dece19549b3876bd795cbdc0f0a P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:10.883605 1338 raft_consensus.cc:513] T 96420dece19549b3876bd795cbdc0f0a P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:10.886425 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96420dece19549b3876bd795cbdc0f0a" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
W20250114 02:54:10.888548 1040 leader_election.cc:343] T 96420dece19549b3876bd795cbdc0f0a P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:10.890908 1340 raft_consensus.cc:2798] T ec3285525faf482d8739f1bedd653503 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:10.892771 471 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_245ce3f7a09e489d92122a15f95bacf7" } new_table_name: "default.Table_bd405598fe054b718a8d6fb5dd875f0e" modify_external_catalogs: true
I20250114 02:54:10.896724 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96420dece19549b3876bd795cbdc0f0a" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:10.897166 1206 ts_tablet_manager.cc:1428] T 96420dece19549b3876bd795cbdc0f0a P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.030s user 0.006s sys 0.001s
I20250114 02:54:10.897303 818 raft_consensus.cc:2463] T 96420dece19549b3876bd795cbdc0f0a P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 0.
I20250114 02:54:10.915529 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec3285525faf482d8739f1bedd653503" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
I20250114 02:54:10.916049 962 raft_consensus.cc:3054] T ec3285525faf482d8739f1bedd653503 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:10.917877 797 tablet_service.cc:1514] Processing DeleteTablet for tablet 96420dece19549b3876bd795cbdc0f0a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:09 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:10.918800 1206 tablet_bootstrap.cc:492] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:10.923058 962 raft_consensus.cc:2463] T ec3285525faf482d8739f1bedd653503 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1.
I20250114 02:54:10.927309 1222 ts_tablet_manager.cc:1918] T 180a3965833b4ac29467e5207e45d500 P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:10.930074 1222 log.cc:1198] T 180a3965833b4ac29467e5207e45d500 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/180a3965833b4ac29467e5207e45d500
I20250114 02:54:10.930042 465 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet 268a12df225e4ba59d4a84c2869d4661
I20250114 02:54:10.931140 1222 ts_tablet_manager.cc:1939] T 180a3965833b4ac29467e5207e45d500 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:10.932865 797 tablet_service.cc:1514] Processing DeleteTablet for tablet 268a12df225e4ba59d4a84c2869d4661 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:10 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:10.933476 936 tablet_service.cc:1514] Processing DeleteTablet for tablet 268a12df225e4ba59d4a84c2869d4661 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:10 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:10.951927 1222 tablet_replica.cc:331] T 96420dece19549b3876bd795cbdc0f0a P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:10.952503 1206 tablet_bootstrap.cc:654] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:10.952651 1222 raft_consensus.cc:2238] T 96420dece19549b3876bd795cbdc0f0a P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:10.953449 1222 raft_consensus.cc:2267] T 96420dece19549b3876bd795cbdc0f0a P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:10.954547 936 tablet_service.cc:1467] Processing CreateTablet for tablet 4bb9b429af92446bbe13894d5dfe4c47 (DEFAULT_TABLE table=default.Table_2a1287091fb9466d9e2fae14499a0963 [id=22b7edcf4b7e4c31b3c23dc44ccf028e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:10.955981 936 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bb9b429af92446bbe13894d5dfe4c47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:10.956693 937 tablet_service.cc:1467] Processing CreateTablet for tablet b63dd90f60a04465b948171e74888e79 (DEFAULT_TABLE table=default.Table_9f87e3d3b1234f389cc6b4f39cdf306e [id=d8c9691818d042d7b49475fa009c4ca1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:10.958170 937 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b63dd90f60a04465b948171e74888e79. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:10.958196 1222 ts_tablet_manager.cc:1905] T 96420dece19549b3876bd795cbdc0f0a P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:10.898593 1042 leader_election.cc:304] T 96420dece19549b3876bd795cbdc0f0a P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters: 525198593c8e47afaee54e74267b87e3
W20250114 02:54:10.965121 1042 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.65:39397, user_credentials={real_user=slave}} blocked reactor thread for 66624.9us
I20250114 02:54:10.965801 1341 raft_consensus.cc:2798] T 96420dece19549b3876bd795cbdc0f0a P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:10.966156 1341 raft_consensus.cc:491] T 96420dece19549b3876bd795cbdc0f0a P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:10.966442 1341 raft_consensus.cc:3054] T 96420dece19549b3876bd795cbdc0f0a P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:10.969913 1080 tablet_service.cc:1514] Processing DeleteTablet for tablet 96420dece19549b3876bd795cbdc0f0a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:09 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:10.970582 1223 tablet_replica.cc:331] T 96420dece19549b3876bd795cbdc0f0a P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:10.976166 1341 raft_consensus.cc:513] T 96420dece19549b3876bd795cbdc0f0a P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:10.977144 1223 raft_consensus.cc:2238] T 96420dece19549b3876bd795cbdc0f0a P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:10.978024 1223 raft_consensus.cc:2267] T 96420dece19549b3876bd795cbdc0f0a P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:10.978180 1206 tablet_bootstrap.cc:492] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:10.978677 1206 ts_tablet_manager.cc:1397] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.060s user 0.007s sys 0.003s
I20250114 02:54:10.979571 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96420dece19549b3876bd795cbdc0f0a" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
W20250114 02:54:10.981222 1040 leader_election.cc:343] T 96420dece19549b3876bd795cbdc0f0a P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:10.982107 1206 raft_consensus.cc:357] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:10.982990 1206 raft_consensus.cc:383] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:10.983888 940 tablet_service.cc:1467] Processing CreateTablet for tablet 83b5b36a26e74668998a19fa0ddf7188 (DEFAULT_TABLE table=default.Table_7a318166ed3045c7bbae5b779be062c4 [id=bb08f9cb85c44d108156675ff51b37c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:10.983307 1341 leader_election.cc:290] T 96420dece19549b3876bd795cbdc0f0a P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Requested vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:10.912865 1309 leader_election.cc:290] T ec3285525faf482d8739f1bedd653503 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Requested vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:10.933012 1084 tablet_service.cc:1514] Processing DeleteTablet for tablet 268a12df225e4ba59d4a84c2869d4661 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:10 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:10.955000 1086 tablet_service.cc:1467] Processing CreateTablet for tablet b63dd90f60a04465b948171e74888e79 (DEFAULT_TABLE table=default.Table_9f87e3d3b1234f389cc6b4f39cdf306e [id=d8c9691818d042d7b49475fa009c4ca1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:10.984922 940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83b5b36a26e74668998a19fa0ddf7188. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:10.985455 1086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b63dd90f60a04465b948171e74888e79. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:10.956732 1082 tablet_service.cc:1467] Processing CreateTablet for tablet 83b5b36a26e74668998a19fa0ddf7188 (DEFAULT_TABLE table=default.Table_7a318166ed3045c7bbae5b779be062c4 [id=bb08f9cb85c44d108156675ff51b37c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:10.988827 1082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83b5b36a26e74668998a19fa0ddf7188. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:10.989260 941 tablet_service.cc:1467] Processing CreateTablet for tablet 268a12df225e4ba59d4a84c2869d4661 (DEFAULT_TABLE table=default.Table_a1731c2cc5964c3aaddb43ec22096aad [id=67870401767c48be838c7163a9ba8beb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:10.991137 941 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 268a12df225e4ba59d4a84c2869d4661. 1 dirs total, 0 dirs full, 0 dirs failed
W20250114 02:54:11.000372 469 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 83b5b36a26e74668998a19fa0ddf7188 consensus state has no leader
I20250114 02:54:11.008445 1340 raft_consensus.cc:695] T ec3285525faf482d8739f1bedd653503 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Becoming Leader. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Running, Role: LEADER
I20250114 02:54:11.009595 1207 tablet_bootstrap.cc:492] T 180a3965833b4ac29467e5207e45d500 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:11.010425 1207 ts_tablet_manager.cc:1397] T 180a3965833b4ac29467e5207e45d500 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.307s user 0.006s sys 0.005s
I20250114 02:54:11.014340 1081 tablet_service.cc:1467] Processing CreateTablet for tablet 268a12df225e4ba59d4a84c2869d4661 (DEFAULT_TABLE table=default.Table_a1731c2cc5964c3aaddb43ec22096aad [id=67870401767c48be838c7163a9ba8beb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:11.015417 1081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 268a12df225e4ba59d4a84c2869d4661. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:11.015780 1207 raft_consensus.cc:357] T 180a3965833b4ac29467e5207e45d500 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:11.016453 1207 raft_consensus.cc:383] T 180a3965833b4ac29467e5207e45d500 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:11.016673 1207 raft_consensus.cc:738] T 180a3965833b4ac29467e5207e45d500 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:11.018990 469 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_67fb7ad07b314b26b85dd43b95ea1faa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:11.020632 469 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_67fb7ad07b314b26b85dd43b95ea1faa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:11.024354 1338 leader_election.cc:290] T 96420dece19549b3876bd795cbdc0f0a P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:11.009194 1340 consensus_queue.cc:237] T ec3285525faf482d8739f1bedd653503 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:11.026629 1223 ts_tablet_manager.cc:1905] T 96420dece19549b3876bd795cbdc0f0a P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:11.027096 1206 raft_consensus.cc:738] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:11.027956 1207 consensus_queue.cc:260] T 180a3965833b4ac29467e5207e45d500 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:11.027689 1206 consensus_queue.cc:260] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:11.041065 1206 ts_tablet_manager.cc:1428] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.062s user 0.009s sys 0.000s
I20250114 02:54:11.041893 1206 tablet_bootstrap.cc:492] T a3886c5f976445cebc95ef1839a74aa9 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:11.060617 1206 tablet_bootstrap.cc:654] T a3886c5f976445cebc95ef1839a74aa9 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:11.067611 1205 tablet_bootstrap.cc:492] T a4dd92b0133e413d8b895a0f33e2faf1 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:11.068051 1205 ts_tablet_manager.cc:1397] T a4dd92b0133e413d8b895a0f33e2faf1 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.433s user 0.019s sys 0.019s
I20250114 02:54:11.070374 1205 raft_consensus.cc:357] T a4dd92b0133e413d8b895a0f33e2faf1 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:11.071028 1205 raft_consensus.cc:383] T a4dd92b0133e413d8b895a0f33e2faf1 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:11.071300 1205 raft_consensus.cc:738] T a4dd92b0133e413d8b895a0f33e2faf1 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:11.074788 1207 ts_tablet_manager.cc:1428] T 180a3965833b4ac29467e5207e45d500 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.064s user 0.007s sys 0.000s
I20250114 02:54:11.080929 1206 tablet_bootstrap.cc:492] T a3886c5f976445cebc95ef1839a74aa9 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:11.081349 1206 ts_tablet_manager.cc:1397] T a3886c5f976445cebc95ef1839a74aa9 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.040s user 0.007s sys 0.003s
I20250114 02:54:11.082600 1223 ts_tablet_manager.cc:1918] T 96420dece19549b3876bd795cbdc0f0a P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:11.083058 1223 log.cc:1198] T 96420dece19549b3876bd795cbdc0f0a P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/96420dece19549b3876bd795cbdc0f0a
I20250114 02:54:11.083426 1206 raft_consensus.cc:357] T a3886c5f976445cebc95ef1839a74aa9 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:11.083899 1223 ts_tablet_manager.cc:1939] T 96420dece19549b3876bd795cbdc0f0a P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
I20250114 02:54:11.081651 1225 ts_tablet_manager.cc:1918] T 180a3965833b4ac29467e5207e45d500 P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:11.084050 1206 raft_consensus.cc:383] T a3886c5f976445cebc95ef1839a74aa9 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:11.084335 1206 raft_consensus.cc:738] T a3886c5f976445cebc95ef1839a74aa9 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:11.084502 1225 log.cc:1198] T 180a3965833b4ac29467e5207e45d500 P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/180a3965833b4ac29467e5207e45d500
I20250114 02:54:11.085253 1225 ts_tablet_manager.cc:1939] T 180a3965833b4ac29467e5207e45d500 P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:11.084973 1206 consensus_queue.cc:260] T a3886c5f976445cebc95ef1839a74aa9 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:11.087596 1207 tablet_bootstrap.cc:492] T a3886c5f976445cebc95ef1839a74aa9 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:11.090169 1309 raft_consensus.cc:491] T ba074d4ea17747d5aa9287bdaf541681 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:11.090790 1309 raft_consensus.cc:513] T ba074d4ea17747d5aa9287bdaf541681 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:11.093802 465 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_9a8160fad0814673b89025ed11bc3fba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:11.097412 465 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9a8160fad0814673b89025ed11bc3fba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:11.101128 464 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_d7f47fe824054bc0add3873c12753fab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:11.105381 1309 leader_election.cc:290] T ba074d4ea17747d5aa9287bdaf541681 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:11.105441 462 catalog_manager.cc:5526] T ec3285525faf482d8739f1bedd653503 P 90d6c2c58bb54e0eafb365625ac33871 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67). New cstate: current_term: 1 leader_uuid: "90d6c2c58bb54e0eafb365625ac33871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: UNKNOWN } } }
W20250114 02:54:11.109346 464 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d7f47fe824054bc0add3873c12753fab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:11.114585 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet 96420dece19549b3876bd795cbdc0f0a (table default.Table_d12c2d9a4b3f42d4839548ca62645693 [id=ceeff15df8714e78b662b1cb4e4fc105]) successfully deleted
W20250114 02:54:11.116077 458 catalog_manager.cc:4839] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 268a12df225e4ba59d4a84c2869d4661 because the tablet was not found. No further retry: Not found: Tablet not found: 268a12df225e4ba59d4a84c2869d4661
I20250114 02:54:11.148933 467 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_6741923eb4f845ddbe5844a0e21b38bf" } modify_external_catalogs: true
I20250114 02:54:11.149612 467 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_6741923eb4f845ddbe5844a0e21b38bf" } modify_external_catalogs: true
W20250114 02:54:11.157927 471 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 179248cb5acf4740b268e0ec8b2852f9 has no consensus state
I20250114 02:54:11.171600 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet 180a3965833b4ac29467e5207e45d500 (table default.Table_97368d1825cf41e981b17a5e7c8d2148 [id=c82d594dc3d94b0981444ed5805e2034]) successfully deleted
I20250114 02:54:11.172158 1225 tablet_replica.cc:331] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:11.172940 1225 raft_consensus.cc:2238] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:11.173358 1225 raft_consensus.cc:2267] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:11.174417 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba074d4ea17747d5aa9287bdaf541681" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:11.175025 1225 ts_tablet_manager.cc:1905] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:11.175236 962 raft_consensus.cc:2388] T ba074d4ea17747d5aa9287bdaf541681 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 90d6c2c58bb54e0eafb365625ac33871 in current term 1: Already voted for candidate 819f9e3412e54d44abb11d8b3ca877c2 in this term.
I20250114 02:54:11.176853 1207 tablet_bootstrap.cc:654] T a3886c5f976445cebc95ef1839a74aa9 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:11.173054 1205 consensus_queue.cc:260] T a4dd92b0133e413d8b895a0f33e2faf1 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:11.191591 1225 ts_tablet_manager.cc:1918] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:11.191939 1225 log.cc:1198] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/c69d5eef3f3f49699e76dcbe1be7d6f6
I20250114 02:54:11.193073 1225 ts_tablet_manager.cc:1939] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:11.195263 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet c69d5eef3f3f49699e76dcbe1be7d6f6 (table default.Table_24dc0a51cadc4387918d43a65986ebf4 [id=0661845e06704345b8498b0060e3a40d]) successfully deleted
W20250114 02:54:11.197436 456 catalog_manager.cc:4852] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet 268a12df225e4ba59d4a84c2869d4661 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 268a12df225e4ba59d4a84c2869d4661 already in progress: creating tablet
I20250114 02:54:11.203480 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet 180a3965833b4ac29467e5207e45d500 (table default.Table_97368d1825cf41e981b17a5e7c8d2148 [id=c82d594dc3d94b0981444ed5805e2034]) successfully deleted
I20250114 02:54:11.208827 471 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_b5367d1b4b2a4ec5a97350612e373c40" } new_table_name: "default.Table_d90b2995b13b4a8fb7e50cc4b79cbf84" modify_external_catalogs: true
I20250114 02:54:11.248139 794 tablet_service.cc:1467] Processing CreateTablet for tablet 83b5b36a26e74668998a19fa0ddf7188 (DEFAULT_TABLE table=default.Table_7a318166ed3045c7bbae5b779be062c4 [id=bb08f9cb85c44d108156675ff51b37c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:11.249603 794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83b5b36a26e74668998a19fa0ddf7188. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:11.257179 796 tablet_service.cc:1467] Processing CreateTablet for tablet b63dd90f60a04465b948171e74888e79 (DEFAULT_TABLE table=default.Table_9f87e3d3b1234f389cc6b4f39cdf306e [id=d8c9691818d042d7b49475fa009c4ca1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:11.258301 796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b63dd90f60a04465b948171e74888e79. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:11.269150 797 tablet_service.cc:1467] Processing CreateTablet for tablet 268a12df225e4ba59d4a84c2869d4661 (DEFAULT_TABLE table=default.Table_a1731c2cc5964c3aaddb43ec22096aad [id=67870401767c48be838c7163a9ba8beb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:11.270215 797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 268a12df225e4ba59d4a84c2869d4661. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:11.280474 1205 ts_tablet_manager.cc:1428] T a4dd92b0133e413d8b895a0f33e2faf1 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.212s user 0.006s sys 0.004s
I20250114 02:54:11.284871 816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba074d4ea17747d5aa9287bdaf541681" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:11.286598 1042 leader_election.cc:304] T ba074d4ea17747d5aa9287bdaf541681 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90d6c2c58bb54e0eafb365625ac33871; no voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2
I20250114 02:54:11.287613 1309 raft_consensus.cc:3054] T ba074d4ea17747d5aa9287bdaf541681 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:11.294555 1309 raft_consensus.cc:2743] T ba074d4ea17747d5aa9287bdaf541681 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:11.295718 817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96420dece19549b3876bd795cbdc0f0a" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
W20250114 02:54:11.297228 1042 leader_election.cc:343] T 96420dece19549b3876bd795cbdc0f0a P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Illegal state: Tablet not RUNNING: STOPPED
I20250114 02:54:11.297780 1042 leader_election.cc:304] T 96420dece19549b3876bd795cbdc0f0a P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90d6c2c58bb54e0eafb365625ac33871; no voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2
W20250114 02:54:11.298309 1042 raft_consensus.cc:2681] T 96420dece19549b3876bd795cbdc0f0a P 90d6c2c58bb54e0eafb365625ac33871: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250114 02:54:11.305510 1205 tablet_bootstrap.cc:492] T 857119ca5f9d4ad98f899df22f32244b P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:11.312925 1206 ts_tablet_manager.cc:1428] T a3886c5f976445cebc95ef1839a74aa9 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.231s user 0.006s sys 0.001s
I20250114 02:54:11.314126 1206 tablet_bootstrap.cc:492] T 857119ca5f9d4ad98f899df22f32244b P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:11.319480 1222 ts_tablet_manager.cc:1918] T 96420dece19549b3876bd795cbdc0f0a P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:11.325196 1222 log.cc:1198] T 96420dece19549b3876bd795cbdc0f0a P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/96420dece19549b3876bd795cbdc0f0a
I20250114 02:54:11.325928 1222 ts_tablet_manager.cc:1939] T 96420dece19549b3876bd795cbdc0f0a P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:11.328027 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet 96420dece19549b3876bd795cbdc0f0a (table default.Table_d12c2d9a4b3f42d4839548ca62645693 [id=ceeff15df8714e78b662b1cb4e4fc105]) successfully deleted
W20250114 02:54:11.330636 458 catalog_manager.cc:4839] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet 268a12df225e4ba59d4a84c2869d4661 because the tablet was not found. No further retry: Not found: Tablet not found: 268a12df225e4ba59d4a84c2869d4661
I20250114 02:54:11.332780 469 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet a3886c5f976445cebc95ef1839a74aa9 (table default.Table_35a78676616e4b20906b262c9d0c0d9e [id=e984149e9da444f1b2802e3f35a60fd6]). Expected version 2 got 0
I20250114 02:54:11.336396 793 tablet_service.cc:1514] Processing DeleteTablet for tablet c69d5eef3f3f49699e76dcbe1be7d6f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:10 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:11.341352 1206 tablet_bootstrap.cc:654] T 857119ca5f9d4ad98f899df22f32244b P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:11.343047 1222 tablet_replica.cc:331] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:11.343868 1222 raft_consensus.cc:2238] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:11.343997 469 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet a3886c5f976445cebc95ef1839a74aa9 (table default.Table_35a78676616e4b20906b262c9d0c0d9e [id=e984149e9da444f1b2802e3f35a60fd6]). Expected version 2 got 0
I20250114 02:54:11.344375 1222 raft_consensus.cc:2267] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:11.346237 1222 ts_tablet_manager.cc:1905] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:11.349735 1207 tablet_bootstrap.cc:492] T a3886c5f976445cebc95ef1839a74aa9 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:11.350164 1207 ts_tablet_manager.cc:1397] T a3886c5f976445cebc95ef1839a74aa9 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.263s user 0.008s sys 0.001s
I20250114 02:54:11.360785 468 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet 34b7a95953f043a08d1bce25d7af4c00
I20250114 02:54:11.366021 1206 tablet_bootstrap.cc:492] T 857119ca5f9d4ad98f899df22f32244b P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:11.366413 1206 ts_tablet_manager.cc:1397] T 857119ca5f9d4ad98f899df22f32244b P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.052s user 0.005s sys 0.007s
I20250114 02:54:11.371330 1206 raft_consensus.cc:357] T 857119ca5f9d4ad98f899df22f32244b P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:11.372035 1206 raft_consensus.cc:383] T 857119ca5f9d4ad98f899df22f32244b P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:11.372349 1206 raft_consensus.cc:738] T 857119ca5f9d4ad98f899df22f32244b P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:11.372895 1206 consensus_queue.cc:260] T 857119ca5f9d4ad98f899df22f32244b P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
W20250114 02:54:11.380187 456 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 179248cb5acf4740b268e0ec8b2852f9 (table default.Table_bd405598fe054b718a8d6fb5dd875f0e [id=5745501ecd7b4cecaca3d35502a6cc40], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 179248cb5acf4740b268e0ec8b2852f9 has no consensus state
I20250114 02:54:11.383376 1081 tablet_service.cc:1514] Processing DeleteTablet for tablet 34b7a95953f043a08d1bce25d7af4c00 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:10 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:11.384068 1223 tablet_replica.cc:331] T 34b7a95953f043a08d1bce25d7af4c00 P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:11.384584 793 tablet_service.cc:1514] Processing DeleteTablet for tablet 34b7a95953f043a08d1bce25d7af4c00 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:10 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:11.384747 1223 raft_consensus.cc:2238] T 34b7a95953f043a08d1bce25d7af4c00 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:11.384950 1206 ts_tablet_manager.cc:1428] T 857119ca5f9d4ad98f899df22f32244b P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.018s user 0.002s sys 0.003s
I20250114 02:54:11.385138 1223 raft_consensus.cc:2267] T 34b7a95953f043a08d1bce25d7af4c00 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:11.386620 1223 ts_tablet_manager.cc:1905] T 34b7a95953f043a08d1bce25d7af4c00 P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:11.388752 1206 tablet_bootstrap.cc:492] T a4dd92b0133e413d8b895a0f33e2faf1 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:11.390384 938 tablet_service.cc:1514] Processing DeleteTablet for tablet 34b7a95953f043a08d1bce25d7af4c00 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:10 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:11.391111 1225 tablet_replica.cc:331] T 34b7a95953f043a08d1bce25d7af4c00 P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:11.391712 1225 raft_consensus.cc:2238] T 34b7a95953f043a08d1bce25d7af4c00 P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Raft consensus shutting down.
I20250114 02:54:11.392325 1225 pending_rounds.cc:62] T 34b7a95953f043a08d1bce25d7af4c00 P 525198593c8e47afaee54e74267b87e3: Trying to abort 1 pending ops.
I20250114 02:54:11.392503 1225 pending_rounds.cc:69] T 34b7a95953f043a08d1bce25d7af4c00 P 525198593c8e47afaee54e74267b87e3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114028034500579328 op_type: NO_OP noop_request { }
I20250114 02:54:11.392808 1225 raft_consensus.cc:2883] T 34b7a95953f043a08d1bce25d7af4c00 P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250114 02:54:11.393100 1225 raft_consensus.cc:2267] T 34b7a95953f043a08d1bce25d7af4c00 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:11.393272 1222 ts_tablet_manager.cc:1918] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:11.393594 1222 log.cc:1198] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/c69d5eef3f3f49699e76dcbe1be7d6f6
I20250114 02:54:11.394161 1222 ts_tablet_manager.cc:1939] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:11.395869 1225 ts_tablet_manager.cc:1905] T 34b7a95953f043a08d1bce25d7af4c00 P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:11.406749 468 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_9f87e3d3b1234f389cc6b4f39cdf306e" } new_table_name: "default.Table_1097f09fb10e441fb639f8d7aa12d487" modify_external_catalogs: true
I20250114 02:54:11.408118 1225 ts_tablet_manager.cc:1918] T 34b7a95953f043a08d1bce25d7af4c00 P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:11.408600 1225 log.cc:1198] T 34b7a95953f043a08d1bce25d7af4c00 P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/34b7a95953f043a08d1bce25d7af4c00
I20250114 02:54:11.409340 1225 ts_tablet_manager.cc:1939] T 34b7a95953f043a08d1bce25d7af4c00 P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:11.411659 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet 34b7a95953f043a08d1bce25d7af4c00 (table default.Table_c99fdc20f32d41baadd11eb425b279ac [id=c14ed2074c144373a0314f99813e8da3]) successfully deleted
I20250114 02:54:11.422626 1081 tablet_service.cc:1514] Processing DeleteTablet for tablet 180a3965833b4ac29467e5207e45d500 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:10 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:11.424304 470 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_02b3fc3bfe484e3a9c5d5e9114deb368"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:11.425688 470 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_02b3fc3bfe484e3a9c5d5e9114deb368 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:11.425858 1205 tablet_bootstrap.cc:654] T 857119ca5f9d4ad98f899df22f32244b P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:11.428928 463 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_95f4830fe0bc41519e53f94f9514583a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:11.429210 1207 raft_consensus.cc:357] T a3886c5f976445cebc95ef1839a74aa9 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:11.429857 1207 raft_consensus.cc:383] T a3886c5f976445cebc95ef1839a74aa9 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:11.430122 1207 raft_consensus.cc:738] T a3886c5f976445cebc95ef1839a74aa9 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
W20250114 02:54:11.430536 463 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_95f4830fe0bc41519e53f94f9514583a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:11.430729 1207 consensus_queue.cc:260] T a3886c5f976445cebc95ef1839a74aa9 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:11.432914 1223 ts_tablet_manager.cc:1918] T 34b7a95953f043a08d1bce25d7af4c00 P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:11.437048 1223 log.cc:1198] T 34b7a95953f043a08d1bce25d7af4c00 P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/34b7a95953f043a08d1bce25d7af4c00
I20250114 02:54:11.438199 1223 ts_tablet_manager.cc:1939] T 34b7a95953f043a08d1bce25d7af4c00 P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
I20250114 02:54:11.441282 1223 tablet_replica.cc:331] T 180a3965833b4ac29467e5207e45d500 P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:11.443931 1207 ts_tablet_manager.cc:1428] T a3886c5f976445cebc95ef1839a74aa9 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.093s user 0.003s sys 0.002s
I20250114 02:54:11.444145 1223 raft_consensus.cc:2238] T 180a3965833b4ac29467e5207e45d500 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:11.444582 1223 raft_consensus.cc:2267] T 180a3965833b4ac29467e5207e45d500 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:11.444819 1207 tablet_bootstrap.cc:492] T a4dd92b0133e413d8b895a0f33e2faf1 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:11.448488 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet 34b7a95953f043a08d1bce25d7af4c00 (table default.Table_c99fdc20f32d41baadd11eb425b279ac [id=c14ed2074c144373a0314f99813e8da3]) successfully deleted
I20250114 02:54:11.454810 938 tablet_service.cc:1514] Processing DeleteTablet for tablet a4dd92b0133e413d8b895a0f33e2faf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:10 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:11.455729 1225 tablet_replica.cc:331] T a4dd92b0133e413d8b895a0f33e2faf1 P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:11.456420 1225 raft_consensus.cc:2238] T a4dd92b0133e413d8b895a0f33e2faf1 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:11.456882 1225 raft_consensus.cc:2267] T a4dd92b0133e413d8b895a0f33e2faf1 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:11.459082 1225 ts_tablet_manager.cc:1905] T a4dd92b0133e413d8b895a0f33e2faf1 P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:11.467075 1207 tablet_bootstrap.cc:654] T a4dd92b0133e413d8b895a0f33e2faf1 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:11.464143 466 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet a3886c5f976445cebc95ef1839a74aa9 (table default.Table_35a78676616e4b20906b262c9d0c0d9e [id=e984149e9da444f1b2802e3f35a60fd6]). Expected version 2 got 0
I20250114 02:54:11.482142 462 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet a3886c5f976445cebc95ef1839a74aa9 (table default.Table_35a78676616e4b20906b262c9d0c0d9e [id=e984149e9da444f1b2802e3f35a60fd6]). Expected version 2 got 0
I20250114 02:54:11.484855 1225 ts_tablet_manager.cc:1918] T a4dd92b0133e413d8b895a0f33e2faf1 P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:11.485184 1225 log.cc:1198] T a4dd92b0133e413d8b895a0f33e2faf1 P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/a4dd92b0133e413d8b895a0f33e2faf1
I20250114 02:54:11.485731 1225 ts_tablet_manager.cc:1939] T a4dd92b0133e413d8b895a0f33e2faf1 P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:11.489354 1206 tablet_bootstrap.cc:654] T a4dd92b0133e413d8b895a0f33e2faf1 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:11.491796 1223 ts_tablet_manager.cc:1905] T 180a3965833b4ac29467e5207e45d500 P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:11.497802 1205 tablet_bootstrap.cc:492] T 857119ca5f9d4ad98f899df22f32244b P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:11.498200 1205 ts_tablet_manager.cc:1397] T 857119ca5f9d4ad98f899df22f32244b P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.193s user 0.014s sys 0.003s
I20250114 02:54:11.500501 1222 tablet_replica.cc:331] T 34b7a95953f043a08d1bce25d7af4c00 P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:11.500419 1205 raft_consensus.cc:357] T 857119ca5f9d4ad98f899df22f32244b P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:11.501045 1205 raft_consensus.cc:383] T 857119ca5f9d4ad98f899df22f32244b P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:11.501137 1222 raft_consensus.cc:2238] T 34b7a95953f043a08d1bce25d7af4c00 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:11.501292 1205 raft_consensus.cc:738] T 857119ca5f9d4ad98f899df22f32244b P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:11.501555 1222 raft_consensus.cc:2267] T 34b7a95953f043a08d1bce25d7af4c00 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:11.502337 1205 consensus_queue.cc:260] T 857119ca5f9d4ad98f899df22f32244b P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:11.503068 1222 ts_tablet_manager.cc:1905] T 34b7a95953f043a08d1bce25d7af4c00 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:11.504678 1207 tablet_bootstrap.cc:492] T a4dd92b0133e413d8b895a0f33e2faf1 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:11.504704 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet c69d5eef3f3f49699e76dcbe1be7d6f6 (table default.Table_24dc0a51cadc4387918d43a65986ebf4 [id=0661845e06704345b8498b0060e3a40d]) successfully deleted
I20250114 02:54:11.505029 1207 ts_tablet_manager.cc:1397] T a4dd92b0133e413d8b895a0f33e2faf1 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.060s user 0.010s sys 0.000s
I20250114 02:54:11.509414 1223 ts_tablet_manager.cc:1918] T 180a3965833b4ac29467e5207e45d500 P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:11.509799 1223 log.cc:1198] T 180a3965833b4ac29467e5207e45d500 P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/180a3965833b4ac29467e5207e45d500
I20250114 02:54:11.510466 1223 ts_tablet_manager.cc:1939] T 180a3965833b4ac29467e5207e45d500 P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
I20250114 02:54:11.512185 1106 raft_consensus.cc:1270] T ba074d4ea17747d5aa9287bdaf541681 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Refusing update from remote peer 819f9e3412e54d44abb11d8b3ca877c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 02:54:11.513269 962 raft_consensus.cc:1270] T ba074d4ea17747d5aa9287bdaf541681 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Refusing update from remote peer 819f9e3412e54d44abb11d8b3ca877c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 02:54:11.513665 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet 180a3965833b4ac29467e5207e45d500 (table default.Table_97368d1825cf41e981b17a5e7c8d2148 [id=c82d594dc3d94b0981444ed5805e2034]) successfully deleted
I20250114 02:54:11.513602 1207 raft_consensus.cc:357] T a4dd92b0133e413d8b895a0f33e2faf1 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:11.514151 1205 ts_tablet_manager.cc:1428] T 857119ca5f9d4ad98f899df22f32244b P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.016s user 0.006s sys 0.001s
I20250114 02:54:11.514381 1207 raft_consensus.cc:383] T a4dd92b0133e413d8b895a0f33e2faf1 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:11.514607 1207 raft_consensus.cc:738] T a4dd92b0133e413d8b895a0f33e2faf1 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:11.514411 1294 consensus_queue.cc:1035] T ba074d4ea17747d5aa9287bdaf541681 P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:54:11.515113 1207 consensus_queue.cc:260] T a4dd92b0133e413d8b895a0f33e2faf1 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:11.515789 1294 consensus_queue.cc:1035] T ba074d4ea17747d5aa9287bdaf541681 P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:11.526574 1205 tablet_bootstrap.cc:492] T 96420dece19549b3876bd795cbdc0f0a P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:11.529287 1207 ts_tablet_manager.cc:1428] T a4dd92b0133e413d8b895a0f33e2faf1 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.024s user 0.005s sys 0.000s
I20250114 02:54:11.537885 1207 tablet_bootstrap.cc:492] T 857119ca5f9d4ad98f899df22f32244b P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:11.538746 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet a4dd92b0133e413d8b895a0f33e2faf1 (table default.Table_075cadc0bfa145718aa9f2f6850f595c [id=ec0fc676edc842b2b7114a4b3fe0626a]) successfully deleted
I20250114 02:54:11.541126 1206 tablet_bootstrap.cc:492] T a4dd92b0133e413d8b895a0f33e2faf1 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:11.541482 1206 ts_tablet_manager.cc:1397] T a4dd92b0133e413d8b895a0f33e2faf1 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.153s user 0.009s sys 0.001s
I20250114 02:54:11.543494 1206 raft_consensus.cc:357] T a4dd92b0133e413d8b895a0f33e2faf1 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:11.544119 1206 raft_consensus.cc:383] T a4dd92b0133e413d8b895a0f33e2faf1 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:11.544366 1206 raft_consensus.cc:738] T a4dd92b0133e413d8b895a0f33e2faf1 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:11.545095 1206 consensus_queue.cc:260] T a4dd92b0133e413d8b895a0f33e2faf1 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:11.546885 1206 ts_tablet_manager.cc:1428] T a4dd92b0133e413d8b895a0f33e2faf1 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250114 02:54:11.553725 1207 tablet_bootstrap.cc:654] T 857119ca5f9d4ad98f899df22f32244b P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:11.556221 1206 tablet_bootstrap.cc:492] T 92b8d48880434ab99eec7f1e452931fa P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:11.560494 1222 ts_tablet_manager.cc:1918] T 34b7a95953f043a08d1bce25d7af4c00 P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:11.560853 1222 log.cc:1198] T 34b7a95953f043a08d1bce25d7af4c00 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/34b7a95953f043a08d1bce25d7af4c00
I20250114 02:54:11.561511 1222 ts_tablet_manager.cc:1939] T 34b7a95953f043a08d1bce25d7af4c00 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:11.565483 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet 34b7a95953f043a08d1bce25d7af4c00 (table default.Table_c99fdc20f32d41baadd11eb425b279ac [id=c14ed2074c144373a0314f99813e8da3]) successfully deleted
I20250114 02:54:11.567512 1205 tablet_bootstrap.cc:654] T 96420dece19549b3876bd795cbdc0f0a P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:11.570960 1206 tablet_bootstrap.cc:654] T 92b8d48880434ab99eec7f1e452931fa P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:11.580997 1207 tablet_bootstrap.cc:492] T 857119ca5f9d4ad98f899df22f32244b P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:11.581367 1207 ts_tablet_manager.cc:1397] T 857119ca5f9d4ad98f899df22f32244b P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.044s user 0.007s sys 0.001s
I20250114 02:54:11.583444 1207 raft_consensus.cc:357] T 857119ca5f9d4ad98f899df22f32244b P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:11.584012 1207 raft_consensus.cc:383] T 857119ca5f9d4ad98f899df22f32244b P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:11.584266 1207 raft_consensus.cc:738] T 857119ca5f9d4ad98f899df22f32244b P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:11.584772 1207 consensus_queue.cc:260] T 857119ca5f9d4ad98f899df22f32244b P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:11.587109 1207 ts_tablet_manager.cc:1428] T 857119ca5f9d4ad98f899df22f32244b P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250114 02:54:11.590641 1207 tablet_bootstrap.cc:492] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:11.597613 1361 tablet.cc:1721] T ba074d4ea17747d5aa9287bdaf541681 P 819f9e3412e54d44abb11d8b3ca877c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250114 02:54:11.598188 464 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_d7f47fe824054bc0add3873c12753fab" } modify_external_catalogs: true
I20250114 02:54:11.598690 464 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_d7f47fe824054bc0add3873c12753fab" } modify_external_catalogs: true
I20250114 02:54:11.600427 1365 tablet.cc:1721] T ba074d4ea17747d5aa9287bdaf541681 P 525198593c8e47afaee54e74267b87e3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250114 02:54:11.600531 1207 tablet_bootstrap.cc:654] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:11.602780 1362 tablet.cc:1721] T ba074d4ea17747d5aa9287bdaf541681 P 90d6c2c58bb54e0eafb365625ac33871: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250114 02:54:11.612005 466 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet ba074d4ea17747d5aa9287bdaf541681 (table default.Table_017604272128419e9c9d1bba85e595cb [id=253dc2a0f94149a78b0f4ab001fba488]). Expected version 1 got 0
I20250114 02:54:11.621248 1207 tablet_bootstrap.cc:492] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:11.621625 1207 ts_tablet_manager.cc:1397] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.031s user 0.009s sys 0.000s
I20250114 02:54:11.623593 1207 raft_consensus.cc:357] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:11.624204 1207 raft_consensus.cc:383] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:11.633599 1207 raft_consensus.cc:738] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:11.634135 1207 consensus_queue.cc:260] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:11.634663 467 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 8b1069674c804bd188d06f2e035b064e
I20250114 02:54:11.642874 1207 ts_tablet_manager.cc:1428] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.021s user 0.005s sys 0.000s
I20250114 02:54:11.656136 1207 tablet_bootstrap.cc:492] T d434cbc9daf549a6bdc04c8c76ead8a6 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:11.665146 1081 tablet_service.cc:1514] Processing DeleteTablet for tablet a4dd92b0133e413d8b895a0f33e2faf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:10 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:11.666014 1223 tablet_replica.cc:331] T a4dd92b0133e413d8b895a0f33e2faf1 P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:11.666676 1223 raft_consensus.cc:2238] T a4dd92b0133e413d8b895a0f33e2faf1 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:11.667119 1223 raft_consensus.cc:2267] T a4dd92b0133e413d8b895a0f33e2faf1 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:11.667160 1205 tablet_bootstrap.cc:492] T 96420dece19549b3876bd795cbdc0f0a P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:11.667529 1205 ts_tablet_manager.cc:1397] T 96420dece19549b3876bd795cbdc0f0a P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.141s user 0.019s sys 0.017s
I20250114 02:54:11.669214 1223 ts_tablet_manager.cc:1905] T a4dd92b0133e413d8b895a0f33e2faf1 P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:11.672279 1205 raft_consensus.cc:357] T 96420dece19549b3876bd795cbdc0f0a P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:11.673125 1205 raft_consensus.cc:383] T 96420dece19549b3876bd795cbdc0f0a P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:11.673470 1205 raft_consensus.cc:738] T 96420dece19549b3876bd795cbdc0f0a P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:11.673619 1207 tablet_bootstrap.cc:654] T d434cbc9daf549a6bdc04c8c76ead8a6 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:11.674187 1205 consensus_queue.cc:260] T 96420dece19549b3876bd795cbdc0f0a P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:11.686915 1205 ts_tablet_manager.cc:1428] T 96420dece19549b3876bd795cbdc0f0a P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.019s user 0.004s sys 0.004s
I20250114 02:54:11.688686 1205 tablet_bootstrap.cc:492] T a3886c5f976445cebc95ef1839a74aa9 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:11.693533 467 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_d1de82caa5b94791b6c7dfed1c376e8a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:11.698719 1205 tablet_bootstrap.cc:654] T a3886c5f976445cebc95ef1839a74aa9 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
W20250114 02:54:11.696070 467 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d1de82caa5b94791b6c7dfed1c376e8a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:11.709604 1207 tablet_bootstrap.cc:492] T d434cbc9daf549a6bdc04c8c76ead8a6 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:11.710979 1207 ts_tablet_manager.cc:1397] T d434cbc9daf549a6bdc04c8c76ead8a6 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.055s user 0.010s sys 0.006s
W20250114 02:54:11.712704 455 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.124:39105, user_credentials={real_user=slave}} blocked reactor thread for 55192.3us
I20250114 02:54:11.714730 1207 raft_consensus.cc:357] T d434cbc9daf549a6bdc04c8c76ead8a6 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:11.715353 1207 raft_consensus.cc:383] T d434cbc9daf549a6bdc04c8c76ead8a6 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:11.715574 1207 raft_consensus.cc:738] T d434cbc9daf549a6bdc04c8c76ead8a6 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:11.716120 1207 consensus_queue.cc:260] T d434cbc9daf549a6bdc04c8c76ead8a6 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:11.722348 1223 ts_tablet_manager.cc:1918] T a4dd92b0133e413d8b895a0f33e2faf1 P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:11.722712 1223 log.cc:1198] T a4dd92b0133e413d8b895a0f33e2faf1 P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/a4dd92b0133e413d8b895a0f33e2faf1
I20250114 02:54:11.723296 1223 ts_tablet_manager.cc:1939] T a4dd92b0133e413d8b895a0f33e2faf1 P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
W20250114 02:54:11.723891 471 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8b4bc535c42e43889401f14b6d59659e consensus state has no leader
I20250114 02:54:11.729519 1205 tablet_bootstrap.cc:492] T a3886c5f976445cebc95ef1839a74aa9 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:11.729876 1205 ts_tablet_manager.cc:1397] T a3886c5f976445cebc95ef1839a74aa9 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.042s user 0.011s sys 0.002s
I20250114 02:54:11.731510 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet a4dd92b0133e413d8b895a0f33e2faf1 (table default.Table_075cadc0bfa145718aa9f2f6850f595c [id=ec0fc676edc842b2b7114a4b3fe0626a]) successfully deleted
I20250114 02:54:11.731948 1205 raft_consensus.cc:357] T a3886c5f976445cebc95ef1839a74aa9 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:11.732409 1294 raft_consensus.cc:491] T a3886c5f976445cebc95ef1839a74aa9 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:11.732609 1205 raft_consensus.cc:383] T a3886c5f976445cebc95ef1839a74aa9 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:11.732867 1294 raft_consensus.cc:513] T a3886c5f976445cebc95ef1839a74aa9 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:11.735011 1294 leader_election.cc:290] T a3886c5f976445cebc95ef1839a74aa9 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:11.737234 961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3886c5f976445cebc95ef1839a74aa9" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:11.732879 1205 raft_consensus.cc:738] T a3886c5f976445cebc95ef1839a74aa9 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:11.739321 1205 consensus_queue.cc:260] T a3886c5f976445cebc95ef1839a74aa9 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:11.739951 1104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3886c5f976445cebc95ef1839a74aa9" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:11.740139 961 raft_consensus.cc:2463] T a3886c5f976445cebc95ef1839a74aa9 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 0.
I20250114 02:54:11.740568 1104 raft_consensus.cc:2463] T a3886c5f976445cebc95ef1839a74aa9 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 0.
I20250114 02:54:11.741797 753 leader_election.cc:304] T a3886c5f976445cebc95ef1839a74aa9 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:11.738885 1206 tablet_bootstrap.cc:492] T 92b8d48880434ab99eec7f1e452931fa P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:11.742937 1206 ts_tablet_manager.cc:1397] T 92b8d48880434ab99eec7f1e452931fa P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.195s user 0.023s sys 0.022s
I20250114 02:54:11.743366 1205 ts_tablet_manager.cc:1428] T a3886c5f976445cebc95ef1839a74aa9 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.013s user 0.006s sys 0.001s
I20250114 02:54:11.743547 1294 raft_consensus.cc:2798] T a3886c5f976445cebc95ef1839a74aa9 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:11.743840 1294 raft_consensus.cc:491] T a3886c5f976445cebc95ef1839a74aa9 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:11.744113 1294 raft_consensus.cc:3054] T a3886c5f976445cebc95ef1839a74aa9 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:11.745779 1205 tablet_bootstrap.cc:492] T d434cbc9daf549a6bdc04c8c76ead8a6 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:11.746687 1206 raft_consensus.cc:357] T 92b8d48880434ab99eec7f1e452931fa P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:11.747292 1206 raft_consensus.cc:383] T 92b8d48880434ab99eec7f1e452931fa P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:11.747534 1206 raft_consensus.cc:738] T 92b8d48880434ab99eec7f1e452931fa P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:11.751390 1205 tablet_bootstrap.cc:654] T d434cbc9daf549a6bdc04c8c76ead8a6 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:11.757998 1207 ts_tablet_manager.cc:1428] T d434cbc9daf549a6bdc04c8c76ead8a6 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.047s user 0.005s sys 0.000s
I20250114 02:54:11.758855 1206 consensus_queue.cc:260] T 92b8d48880434ab99eec7f1e452931fa P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:11.762058 1205 tablet_bootstrap.cc:492] T d434cbc9daf549a6bdc04c8c76ead8a6 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:11.762421 1205 ts_tablet_manager.cc:1397] T d434cbc9daf549a6bdc04c8c76ead8a6 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.009s
I20250114 02:54:11.764828 1235 raft_consensus.cc:491] T 96420dece19549b3876bd795cbdc0f0a P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:11.765240 1235 raft_consensus.cc:513] T 96420dece19549b3876bd795cbdc0f0a P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:11.766227 1206 ts_tablet_manager.cc:1428] T 92b8d48880434ab99eec7f1e452931fa P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.023s user 0.006s sys 0.000s
I20250114 02:54:11.766938 1235 leader_election.cc:290] T 96420dece19549b3876bd795cbdc0f0a P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:11.767066 1206 tablet_bootstrap.cc:492] T d434cbc9daf549a6bdc04c8c76ead8a6 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:11.766831 1294 raft_consensus.cc:513] T a3886c5f976445cebc95ef1839a74aa9 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:11.769788 1294 leader_election.cc:290] T a3886c5f976445cebc95ef1839a74aa9 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Requested vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
W20250114 02:54:11.772032 468 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b63dd90f60a04465b948171e74888e79 consensus state has no leader
I20250114 02:54:11.777618 1207 tablet_bootstrap.cc:492] T 92b8d48880434ab99eec7f1e452931fa P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:11.779610 1104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3886c5f976445cebc95ef1839a74aa9" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:11.780186 1104 raft_consensus.cc:3054] T a3886c5f976445cebc95ef1839a74aa9 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
W20250114 02:54:11.785249 898 leader_election.cc:343] T 96420dece19549b3876bd795cbdc0f0a P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Not found: Tablet not found: 96420dece19549b3876bd795cbdc0f0a
W20250114 02:54:11.786346 898 leader_election.cc:343] T 96420dece19549b3876bd795cbdc0f0a P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Not found: Tablet not found: 96420dece19549b3876bd795cbdc0f0a
I20250114 02:54:11.786685 898 leader_election.cc:304] T 96420dece19549b3876bd795cbdc0f0a P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 525198593c8e47afaee54e74267b87e3; no voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:11.788074 961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3886c5f976445cebc95ef1839a74aa9" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
I20250114 02:54:11.788604 961 raft_consensus.cc:3054] T a3886c5f976445cebc95ef1839a74aa9 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:11.792886 1235 raft_consensus.cc:2743] T 96420dece19549b3876bd795cbdc0f0a P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:11.797179 1207 tablet_bootstrap.cc:654] T 92b8d48880434ab99eec7f1e452931fa P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:11.800724 1206 tablet_bootstrap.cc:654] T d434cbc9daf549a6bdc04c8c76ead8a6 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:11.803378 1104 raft_consensus.cc:2463] T a3886c5f976445cebc95ef1839a74aa9 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 1.
I20250114 02:54:11.804597 753 leader_election.cc:304] T a3886c5f976445cebc95ef1839a74aa9 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:11.805410 1294 raft_consensus.cc:2798] T a3886c5f976445cebc95ef1839a74aa9 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:11.805799 1294 raft_consensus.cc:695] T a3886c5f976445cebc95ef1839a74aa9 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 LEADER]: Becoming Leader. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Running, Role: LEADER
I20250114 02:54:11.806608 1294 consensus_queue.cc:237] T a3886c5f976445cebc95ef1839a74aa9 P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:11.811830 1355 raft_consensus.cc:491] T 857119ca5f9d4ad98f899df22f32244b P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:11.812256 1355 raft_consensus.cc:513] T 857119ca5f9d4ad98f899df22f32244b P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:11.813009 961 raft_consensus.cc:2463] T a3886c5f976445cebc95ef1839a74aa9 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 1.
I20250114 02:54:11.814231 1355 leader_election.cc:290] T 857119ca5f9d4ad98f899df22f32244b P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:11.815753 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "857119ca5f9d4ad98f899df22f32244b" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:11.816385 962 raft_consensus.cc:2463] T 857119ca5f9d4ad98f899df22f32244b P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 0.
I20250114 02:54:11.816716 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "857119ca5f9d4ad98f899df22f32244b" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:11.817314 1106 raft_consensus.cc:2463] T 857119ca5f9d4ad98f899df22f32244b P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 0.
I20250114 02:54:11.817992 751 leader_election.cc:304] T 857119ca5f9d4ad98f899df22f32244b P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
I20250114 02:54:11.818809 1205 raft_consensus.cc:357] T d434cbc9daf549a6bdc04c8c76ead8a6 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:11.819321 1355 raft_consensus.cc:2798] T 857119ca5f9d4ad98f899df22f32244b P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:11.819468 1205 raft_consensus.cc:383] T d434cbc9daf549a6bdc04c8c76ead8a6 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:11.819645 1355 raft_consensus.cc:491] T 857119ca5f9d4ad98f899df22f32244b P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:11.819933 1355 raft_consensus.cc:3054] T 857119ca5f9d4ad98f899df22f32244b P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:11.819695 1205 raft_consensus.cc:738] T d434cbc9daf549a6bdc04c8c76ead8a6 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:11.820544 1206 tablet_bootstrap.cc:492] T d434cbc9daf549a6bdc04c8c76ead8a6 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:11.820936 1206 ts_tablet_manager.cc:1397] T d434cbc9daf549a6bdc04c8c76ead8a6 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.054s user 0.011s sys 0.000s
I20250114 02:54:11.821595 1205 consensus_queue.cc:260] T d434cbc9daf549a6bdc04c8c76ead8a6 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:11.823154 1206 raft_consensus.cc:357] T d434cbc9daf549a6bdc04c8c76ead8a6 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:11.823810 1206 raft_consensus.cc:383] T d434cbc9daf549a6bdc04c8c76ead8a6 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:11.825672 1309 raft_consensus.cc:491] T d434cbc9daf549a6bdc04c8c76ead8a6 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:11.826110 1309 raft_consensus.cc:513] T d434cbc9daf549a6bdc04c8c76ead8a6 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:11.826972 1205 ts_tablet_manager.cc:1428] T d434cbc9daf549a6bdc04c8c76ead8a6 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.064s user 0.006s sys 0.000s
I20250114 02:54:11.827944 1309 leader_election.cc:290] T d434cbc9daf549a6bdc04c8c76ead8a6 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:11.824069 1206 raft_consensus.cc:738] T d434cbc9daf549a6bdc04c8c76ead8a6 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:11.829679 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d434cbc9daf549a6bdc04c8c76ead8a6" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:11.829703 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d434cbc9daf549a6bdc04c8c76ead8a6" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:11.830278 962 raft_consensus.cc:2463] T d434cbc9daf549a6bdc04c8c76ead8a6 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 0.
I20250114 02:54:11.830034 1206 consensus_queue.cc:260] T d434cbc9daf549a6bdc04c8c76ead8a6 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:11.830763 1205 tablet_bootstrap.cc:492] T 92b8d48880434ab99eec7f1e452931fa P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:11.831049 818 raft_consensus.cc:2463] T d434cbc9daf549a6bdc04c8c76ead8a6 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 0.
I20250114 02:54:11.832748 1040 leader_election.cc:304] T d434cbc9daf549a6bdc04c8c76ead8a6 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:11.833770 1309 raft_consensus.cc:2798] T d434cbc9daf549a6bdc04c8c76ead8a6 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:11.834105 1309 raft_consensus.cc:491] T d434cbc9daf549a6bdc04c8c76ead8a6 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:11.834442 1309 raft_consensus.cc:3054] T d434cbc9daf549a6bdc04c8c76ead8a6 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:11.836988 1355 raft_consensus.cc:513] T 857119ca5f9d4ad98f899df22f32244b P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:11.839643 1205 tablet_bootstrap.cc:654] T 92b8d48880434ab99eec7f1e452931fa P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:11.840939 464 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 37e61097bfbb410caa1f3ab2bb0f92f3
I20250114 02:54:11.843115 1309 raft_consensus.cc:513] T d434cbc9daf549a6bdc04c8c76ead8a6 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:11.850414 1207 tablet_bootstrap.cc:492] T 92b8d48880434ab99eec7f1e452931fa P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:11.850966 1207 ts_tablet_manager.cc:1397] T 92b8d48880434ab99eec7f1e452931fa P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.073s user 0.007s sys 0.001s
I20250114 02:54:11.859175 938 tablet_service.cc:1467] Processing CreateTablet for tablet ed2d137579b8467da5eed4ea2f619a9a (DEFAULT_TABLE table=default.Table_1f08339549de49f5b5abc889a248657c [id=0d173fdb502f482fb86b6d3e5fdc66f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:11.860669 938 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed2d137579b8467da5eed4ea2f619a9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:11.864765 1309 leader_election.cc:290] T d434cbc9daf549a6bdc04c8c76ead8a6 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Requested vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:11.868841 940 tablet_service.cc:1467] Processing CreateTablet for tablet 179248cb5acf4740b268e0ec8b2852f9 (DEFAULT_TABLE table=default.Table_bd405598fe054b718a8d6fb5dd875f0e [id=5745501ecd7b4cecaca3d35502a6cc40]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:11.869946 940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 179248cb5acf4740b268e0ec8b2852f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:11.875797 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d434cbc9daf549a6bdc04c8c76ead8a6" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
I20250114 02:54:11.876176 941 tablet_service.cc:1467] Processing CreateTablet for tablet 9bfd8fd6d2104fb4a6484cb8c98981bd (DEFAULT_TABLE table=default.Table_6c885213c7cb45e2b12955642171daff [id=2688bb1e2f4f4765986f738256d17f78]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:11.876520 962 raft_consensus.cc:3054] T d434cbc9daf549a6bdc04c8c76ead8a6 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:11.877729 941 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bfd8fd6d2104fb4a6484cb8c98981bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:11.870500 1207 raft_consensus.cc:357] T 92b8d48880434ab99eec7f1e452931fa P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:11.880951 1207 raft_consensus.cc:383] T 92b8d48880434ab99eec7f1e452931fa P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:11.880862 936 tablet_service.cc:1467] Processing CreateTablet for tablet 9589afd0094545c1801b7a0b5eb8698f (DEFAULT_TABLE table=default.Table_c6ab81139a97463d81e5a7f6334a4006 [id=aad0de2f82d744ce9054dc12a6b55536]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:11.881198 1207 raft_consensus.cc:738] T 92b8d48880434ab99eec7f1e452931fa P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:11.881812 936 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9589afd0094545c1801b7a0b5eb8698f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:11.881807 1081 tablet_service.cc:1467] Processing CreateTablet for tablet ed2d137579b8467da5eed4ea2f619a9a (DEFAULT_TABLE table=default.Table_1f08339549de49f5b5abc889a248657c [id=0d173fdb502f482fb86b6d3e5fdc66f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:11.883163 1081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed2d137579b8467da5eed4ea2f619a9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:11.888505 1207 consensus_queue.cc:260] T 92b8d48880434ab99eec7f1e452931fa P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:11.896020 1207 ts_tablet_manager.cc:1428] T 92b8d48880434ab99eec7f1e452931fa P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.045s user 0.007s sys 0.001s
I20250114 02:54:11.897464 1086 tablet_service.cc:1467] Processing CreateTablet for tablet 179248cb5acf4740b268e0ec8b2852f9 (DEFAULT_TABLE table=default.Table_bd405598fe054b718a8d6fb5dd875f0e [id=5745501ecd7b4cecaca3d35502a6cc40]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:11.904932 1086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 179248cb5acf4740b268e0ec8b2852f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:11.906023 1207 tablet_bootstrap.cc:492] T b4740f6b3a6741368ba68907ad065279 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:11.907852 937 tablet_service.cc:1467] Processing CreateTablet for tablet c20aca761313498193e1d79c10705f8f (DEFAULT_TABLE table=default.Table_6a4fe39298514187b65def5841d3a992 [id=5c5a7974ad2f4e58ba5c47a6eeaaac62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:11.908967 937 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c20aca761313498193e1d79c10705f8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:11.909953 1083 tablet_service.cc:1467] Processing CreateTablet for tablet 9589afd0094545c1801b7a0b5eb8698f (DEFAULT_TABLE table=default.Table_c6ab81139a97463d81e5a7f6334a4006 [id=aad0de2f82d744ce9054dc12a6b55536]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:11.910940 1083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9589afd0094545c1801b7a0b5eb8698f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:11.910720 1085 tablet_service.cc:1467] Processing CreateTablet for tablet c20aca761313498193e1d79c10705f8f (DEFAULT_TABLE table=default.Table_6a4fe39298514187b65def5841d3a992 [id=5c5a7974ad2f4e58ba5c47a6eeaaac62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:11.913030 1085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c20aca761313498193e1d79c10705f8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:11.897670 1082 tablet_service.cc:1467] Processing CreateTablet for tablet 9bfd8fd6d2104fb4a6484cb8c98981bd (DEFAULT_TABLE table=default.Table_6c885213c7cb45e2b12955642171daff [id=2688bb1e2f4f4765986f738256d17f78]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:11.916548 470 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_2217d3b1dcf4455d9103cad045518573"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:11.917361 962 raft_consensus.cc:2463] T d434cbc9daf549a6bdc04c8c76ead8a6 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1.
W20250114 02:54:11.917822 470 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2217d3b1dcf4455d9103cad045518573 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:11.918316 1040 leader_election.cc:304] T d434cbc9daf549a6bdc04c8c76ead8a6 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:11.918982 1309 raft_consensus.cc:2798] T d434cbc9daf549a6bdc04c8c76ead8a6 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:11.919360 1309 raft_consensus.cc:695] T d434cbc9daf549a6bdc04c8c76ead8a6 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Becoming Leader. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Running, Role: LEADER
I20250114 02:54:11.920032 1309 consensus_queue.cc:237] T d434cbc9daf549a6bdc04c8c76ead8a6 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:11.920773 1082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bfd8fd6d2104fb4a6484cb8c98981bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:11.925419 1207 tablet_bootstrap.cc:654] T b4740f6b3a6741368ba68907ad065279 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:11.925516 464 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_9a8160fad0814673b89025ed11bc3fba" } modify_external_catalogs: true
I20250114 02:54:11.926012 464 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_9a8160fad0814673b89025ed11bc3fba" } modify_external_catalogs: true
I20250114 02:54:11.931402 467 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_c6ab81139a97463d81e5a7f6334a4006" } new_table_name: "default.Table_ed523e66ce6945a7a0fcdb96e498131d" modify_external_catalogs: true
I20250114 02:54:11.932857 469 catalog_manager.cc:5526] T a3886c5f976445cebc95ef1839a74aa9 P 819f9e3412e54d44abb11d8b3ca877c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65). New cstate: current_term: 1 leader_uuid: "819f9e3412e54d44abb11d8b3ca877c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:11.837855 1206 ts_tablet_manager.cc:1428] T d434cbc9daf549a6bdc04c8c76ead8a6 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20250114 02:54:11.857589 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d434cbc9daf549a6bdc04c8c76ead8a6" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:11.938256 818 raft_consensus.cc:3054] T d434cbc9daf549a6bdc04c8c76ead8a6 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:11.939355 1206 tablet_bootstrap.cc:492] T b4740f6b3a6741368ba68907ad065279 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:11.943496 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "857119ca5f9d4ad98f899df22f32244b" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
I20250114 02:54:11.943782 939 tablet_service.cc:1467] Processing CreateTablet for tablet 25e85b0c5c9a4a3ca35a001359f21167 (DEFAULT_TABLE table=default.Table_bcf55148b6bc413e8d4e8e31996ff32b [id=d6c16b93fcf5427f9876082d1996a31f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:11.858970 794 tablet_service.cc:1467] Processing CreateTablet for tablet ed2d137579b8467da5eed4ea2f619a9a (DEFAULT_TABLE table=default.Table_1f08339549de49f5b5abc889a248657c [id=0d173fdb502f482fb86b6d3e5fdc66f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:11.944037 962 raft_consensus.cc:3054] T 857119ca5f9d4ad98f899df22f32244b P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:11.944828 939 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25e85b0c5c9a4a3ca35a001359f21167. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:11.944829 794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed2d137579b8467da5eed4ea2f619a9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:11.867838 796 tablet_service.cc:1467] Processing CreateTablet for tablet 179248cb5acf4740b268e0ec8b2852f9 (DEFAULT_TABLE table=default.Table_bd405598fe054b718a8d6fb5dd875f0e [id=5745501ecd7b4cecaca3d35502a6cc40]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:11.871671 797 tablet_service.cc:1467] Processing CreateTablet for tablet 9bfd8fd6d2104fb4a6484cb8c98981bd (DEFAULT_TABLE table=default.Table_6c885213c7cb45e2b12955642171daff [id=2688bb1e2f4f4765986f738256d17f78]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:11.875541 792 tablet_service.cc:1467] Processing CreateTablet for tablet 9589afd0094545c1801b7a0b5eb8698f (DEFAULT_TABLE table=default.Table_c6ab81139a97463d81e5a7f6334a4006 [id=aad0de2f82d744ce9054dc12a6b55536]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:11.908030 793 tablet_service.cc:1467] Processing CreateTablet for tablet c20aca761313498193e1d79c10705f8f (DEFAULT_TABLE table=default.Table_6a4fe39298514187b65def5841d3a992 [id=5c5a7974ad2f4e58ba5c47a6eeaaac62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:11.947888 798 tablet_service.cc:1467] Processing CreateTablet for tablet 25e85b0c5c9a4a3ca35a001359f21167 (DEFAULT_TABLE table=default.Table_bcf55148b6bc413e8d4e8e31996ff32b [id=d6c16b93fcf5427f9876082d1996a31f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:11.948926 798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25e85b0c5c9a4a3ca35a001359f21167. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:11.949846 792 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9589afd0094545c1801b7a0b5eb8698f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:11.950469 1355 leader_election.cc:290] T 857119ca5f9d4ad98f899df22f32244b P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Requested vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:11.950955 797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bfd8fd6d2104fb4a6484cb8c98981bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:11.952203 796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 179248cb5acf4740b268e0ec8b2852f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:11.953212 793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c20aca761313498193e1d79c10705f8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:11.953620 1206 tablet_bootstrap.cc:654] T b4740f6b3a6741368ba68907ad065279 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:11.960536 469 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet a3886c5f976445cebc95ef1839a74aa9 (table default.Table_35a78676616e4b20906b262c9d0c0d9e [id=e984149e9da444f1b2802e3f35a60fd6]). Expected version 2 got 0
I20250114 02:54:11.968442 1084 tablet_service.cc:1467] Processing CreateTablet for tablet 25e85b0c5c9a4a3ca35a001359f21167 (DEFAULT_TABLE table=default.Table_bcf55148b6bc413e8d4e8e31996ff32b [id=d6c16b93fcf5427f9876082d1996a31f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:11.969533 1084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25e85b0c5c9a4a3ca35a001359f21167. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:11.968464 1104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "857119ca5f9d4ad98f899df22f32244b" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:11.971338 1104 raft_consensus.cc:3054] T 857119ca5f9d4ad98f899df22f32244b P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:11.976418 1309 consensus_queue.cc:1035] T ec3285525faf482d8739f1bedd653503 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:54:11.977154 468 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_c6a60d505816422d91eccbe4813d7843"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:11.978832 468 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c6a60d505816422d91eccbe4813d7843 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:11.981730 1205 tablet_bootstrap.cc:492] T 92b8d48880434ab99eec7f1e452931fa P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:11.982293 471 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_1f08339549de49f5b5abc889a248657c" } new_table_name: "default.Table_4c3843233dda407395ab9652d5870bc6" modify_external_catalogs: true
I20250114 02:54:11.982180 463 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_494541d472dc48d0bef33a634c1180fa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:11.983783 463 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_494541d472dc48d0bef33a634c1180fa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:11.991035 1104 raft_consensus.cc:2463] T 857119ca5f9d4ad98f899df22f32244b P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 1.
I20250114 02:54:11.982113 1205 ts_tablet_manager.cc:1397] T 92b8d48880434ab99eec7f1e452931fa P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.152s user 0.007s sys 0.014s
I20250114 02:54:11.998703 1205 raft_consensus.cc:357] T 92b8d48880434ab99eec7f1e452931fa P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:11.999439 1205 raft_consensus.cc:383] T 92b8d48880434ab99eec7f1e452931fa P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:11.999684 1205 raft_consensus.cc:738] T 92b8d48880434ab99eec7f1e452931fa P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:12.004510 753 leader_election.cc:304] T 857119ca5f9d4ad98f899df22f32244b P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:12.013840 1309 consensus_queue.cc:1035] T ec3285525faf482d8739f1bedd653503 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:54:12.015444 1376 raft_consensus.cc:491] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:12.015872 1376 raft_consensus.cc:513] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:12.017731 1376 leader_election.cc:290] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
W20250114 02:54:12.019819 1042 leader_election.cc:343] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Not found: Tablet not found: c69d5eef3f3f49699e76dcbe1be7d6f6
W20250114 02:54:12.022091 1040 leader_election.cc:343] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Not found: Tablet not found: c69d5eef3f3f49699e76dcbe1be7d6f6
I20250114 02:54:12.022522 1040 leader_election.cc:304] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90d6c2c58bb54e0eafb365625ac33871; no voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2
I20250114 02:54:12.023977 1376 raft_consensus.cc:2743] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:12.026270 962 raft_consensus.cc:2463] T 857119ca5f9d4ad98f899df22f32244b P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 1.
I20250114 02:54:12.043993 1205 consensus_queue.cc:260] T 92b8d48880434ab99eec7f1e452931fa P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:12.052244 462 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_e691c983a1bc4ce3a3d4ac683c048218" } modify_external_catalogs: true
I20250114 02:54:12.052812 462 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_e691c983a1bc4ce3a3d4ac683c048218" } modify_external_catalogs: true
I20250114 02:54:12.056267 1205 ts_tablet_manager.cc:1428] T 92b8d48880434ab99eec7f1e452931fa P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.060s user 0.006s sys 0.000s
I20250114 02:54:12.061754 1205 tablet_bootstrap.cc:492] T b4740f6b3a6741368ba68907ad065279 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:12.027622 818 raft_consensus.cc:2463] T d434cbc9daf549a6bdc04c8c76ead8a6 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1.
I20250114 02:54:12.071539 1205 tablet_bootstrap.cc:654] T b4740f6b3a6741368ba68907ad065279 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:12.083137 1207 tablet_bootstrap.cc:492] T b4740f6b3a6741368ba68907ad065279 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:12.083527 1207 ts_tablet_manager.cc:1397] T b4740f6b3a6741368ba68907ad065279 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.178s user 0.002s sys 0.010s
I20250114 02:54:12.098831 1207 raft_consensus.cc:357] T b4740f6b3a6741368ba68907ad065279 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:12.099509 1207 raft_consensus.cc:383] T b4740f6b3a6741368ba68907ad065279 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:12.104697 1207 raft_consensus.cc:738] T b4740f6b3a6741368ba68907ad065279 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
W20250114 02:54:12.105657 753 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.67:45749, user_credentials={real_user=slave}} blocked reactor thread for 101223us
I20250114 02:54:12.107146 1294 raft_consensus.cc:2798] T 857119ca5f9d4ad98f899df22f32244b P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:12.107627 1294 raft_consensus.cc:695] T 857119ca5f9d4ad98f899df22f32244b P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 LEADER]: Becoming Leader. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Running, Role: LEADER
W20250114 02:54:12.110229 467 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9589afd0094545c1801b7a0b5eb8698f consensus state has no leader
I20250114 02:54:12.106590 1207 consensus_queue.cc:260] T b4740f6b3a6741368ba68907ad065279 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:12.108459 1294 consensus_queue.cc:237] T 857119ca5f9d4ad98f899df22f32244b P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:12.116572 464 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 34a9b0c751354484815845e9679e2135
I20250114 02:54:12.118636 1207 ts_tablet_manager.cc:1428] T b4740f6b3a6741368ba68907ad065279 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.035s user 0.002s sys 0.005s
W20250114 02:54:12.127066 471 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ed2d137579b8467da5eed4ea2f619a9a consensus state has no leader
I20250114 02:54:12.132951 465 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet a3886c5f976445cebc95ef1839a74aa9 (table default.Table_35a78676616e4b20906b262c9d0c0d9e [id=e984149e9da444f1b2802e3f35a60fd6]). Expected version 2 got 0
I20250114 02:54:12.134812 466 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: default.Table_017604272128419e9c9d1bba85e595cb [id=253dc2a0f94149a78b0f4ab001fba488] alter complete (version 1)
I20250114 02:54:12.136516 1207 tablet_bootstrap.cc:492] T 61b4b6d4ac364a14a34733b4dd715bb3 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:12.137617 1084 tablet_service.cc:1514] Processing DeleteTablet for tablet a4dd92b0133e413d8b895a0f33e2faf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:10 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:12.142544 1207 tablet_bootstrap.cc:654] T 61b4b6d4ac364a14a34733b4dd715bb3 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
W20250114 02:54:12.148972 458 catalog_manager.cc:4839] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet a4dd92b0133e413d8b895a0f33e2faf1 because the tablet was not found. No further retry: Not found: Tablet not found: a4dd92b0133e413d8b895a0f33e2faf1
I20250114 02:54:12.150411 1084 tablet_service.cc:1514] Processing DeleteTablet for tablet c69d5eef3f3f49699e76dcbe1be7d6f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:10 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:12.151064 1223 tablet_replica.cc:331] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:12.151695 1223 raft_consensus.cc:2238] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:12.152086 1223 raft_consensus.cc:2267] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:12.153592 1223 ts_tablet_manager.cc:1905] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:12.164402 1207 tablet_bootstrap.cc:492] T 61b4b6d4ac364a14a34733b4dd715bb3 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:12.164804 1207 ts_tablet_manager.cc:1397] T 61b4b6d4ac364a14a34733b4dd715bb3 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.046s user 0.009s sys 0.000s
I20250114 02:54:12.165248 940 tablet_service.cc:1514] Processing DeleteTablet for tablet 96420dece19549b3876bd795cbdc0f0a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:09 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:12.166966 1207 raft_consensus.cc:357] T 61b4b6d4ac364a14a34733b4dd715bb3 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:12.167560 1207 raft_consensus.cc:383] T 61b4b6d4ac364a14a34733b4dd715bb3 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:12.167807 1207 raft_consensus.cc:738] T 61b4b6d4ac364a14a34733b4dd715bb3 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:12.168383 1207 consensus_queue.cc:260] T 61b4b6d4ac364a14a34733b4dd715bb3 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:12.173002 464 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_1034610abe1c4464ac9f9ebfd932ef09"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:12.174587 464 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1034610abe1c4464ac9f9ebfd932ef09 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:12.186043 1207 ts_tablet_manager.cc:1428] T 61b4b6d4ac364a14a34733b4dd715bb3 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20250114 02:54:12.189364 1207 tablet_bootstrap.cc:492] T 0752ab4b7031402db4e6be8b61cf364f P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:12.192999 796 tablet_service.cc:1514] Processing DeleteTablet for tablet a4dd92b0133e413d8b895a0f33e2faf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:10 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:12.202454 1207 tablet_bootstrap.cc:654] T 0752ab4b7031402db4e6be8b61cf364f P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:12.212401 1206 tablet_bootstrap.cc:492] T b4740f6b3a6741368ba68907ad065279 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:12.212801 1206 ts_tablet_manager.cc:1397] T b4740f6b3a6741368ba68907ad065279 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.274s user 0.017s sys 0.007s
I20250114 02:54:12.214450 1205 tablet_bootstrap.cc:492] T b4740f6b3a6741368ba68907ad065279 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:12.215539 1205 ts_tablet_manager.cc:1397] T b4740f6b3a6741368ba68907ad065279 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.154s user 0.011s sys 0.037s
I20250114 02:54:12.218590 1205 raft_consensus.cc:357] T b4740f6b3a6741368ba68907ad065279 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:12.222640 1223 ts_tablet_manager.cc:1918] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:12.222996 1223 log.cc:1198] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/c69d5eef3f3f49699e76dcbe1be7d6f6
I20250114 02:54:12.223632 1223 ts_tablet_manager.cc:1939] T c69d5eef3f3f49699e76dcbe1be7d6f6 P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
I20250114 02:54:12.226876 1205 raft_consensus.cc:383] T b4740f6b3a6741368ba68907ad065279 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:12.227821 1205 raft_consensus.cc:738] T b4740f6b3a6741368ba68907ad065279 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:12.230057 1205 consensus_queue.cc:260] T b4740f6b3a6741368ba68907ad065279 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:12.234395 1205 ts_tablet_manager.cc:1428] T b4740f6b3a6741368ba68907ad065279 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.018s user 0.003s sys 0.001s
I20250114 02:54:12.235267 1207 tablet_bootstrap.cc:492] T 0752ab4b7031402db4e6be8b61cf364f P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:12.235723 1207 ts_tablet_manager.cc:1397] T 0752ab4b7031402db4e6be8b61cf364f P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.049s user 0.010s sys 0.000s
I20250114 02:54:12.235919 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet c69d5eef3f3f49699e76dcbe1be7d6f6 (table default.Table_24dc0a51cadc4387918d43a65986ebf4 [id=0661845e06704345b8498b0060e3a40d]) successfully deleted
I20250114 02:54:12.239236 1205 tablet_bootstrap.cc:492] T 0752ab4b7031402db4e6be8b61cf364f P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:12.238085 1207 raft_consensus.cc:357] T 0752ab4b7031402db4e6be8b61cf364f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:12.246019 1207 raft_consensus.cc:383] T 0752ab4b7031402db4e6be8b61cf364f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:12.246282 1207 raft_consensus.cc:738] T 0752ab4b7031402db4e6be8b61cf364f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:12.246886 1207 consensus_queue.cc:260] T 0752ab4b7031402db4e6be8b61cf364f P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:12.254537 1205 tablet_bootstrap.cc:654] T 0752ab4b7031402db4e6be8b61cf364f P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:12.257222 1207 ts_tablet_manager.cc:1428] T 0752ab4b7031402db4e6be8b61cf364f P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.021s user 0.003s sys 0.002s
I20250114 02:54:12.269695 1393 tablet_replica.cc:331] T 96420dece19549b3876bd795cbdc0f0a P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:12.270963 1393 raft_consensus.cc:2238] T 96420dece19549b3876bd795cbdc0f0a P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:12.271649 1393 raft_consensus.cc:2267] T 96420dece19549b3876bd795cbdc0f0a P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:12.273559 1207 tablet_bootstrap.cc:492] T 8b4bc535c42e43889401f14b6d59659e P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:12.278556 1206 raft_consensus.cc:357] T b4740f6b3a6741368ba68907ad065279 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:12.286129 1207 tablet_bootstrap.cc:654] T 8b4bc535c42e43889401f14b6d59659e P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:12.287220 1206 raft_consensus.cc:383] T b4740f6b3a6741368ba68907ad065279 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:12.288410 1206 raft_consensus.cc:738] T b4740f6b3a6741368ba68907ad065279 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:12.290014 1393 ts_tablet_manager.cc:1905] T 96420dece19549b3876bd795cbdc0f0a P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:12.291101 1394 tablet_replica.cc:331] T a4dd92b0133e413d8b895a0f33e2faf1 P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:12.291867 1394 raft_consensus.cc:2238] T a4dd92b0133e413d8b895a0f33e2faf1 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:12.292411 1394 raft_consensus.cc:2267] T a4dd92b0133e413d8b895a0f33e2faf1 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:12.290860 1206 consensus_queue.cc:260] T b4740f6b3a6741368ba68907ad065279 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:12.295310 1205 tablet_bootstrap.cc:492] T 0752ab4b7031402db4e6be8b61cf364f P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:12.299822 1205 ts_tablet_manager.cc:1397] T 0752ab4b7031402db4e6be8b61cf364f P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.061s user 0.024s sys 0.008s
I20250114 02:54:12.301527 1206 ts_tablet_manager.cc:1428] T b4740f6b3a6741368ba68907ad065279 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.088s user 0.002s sys 0.004s
I20250114 02:54:12.308635 1206 tablet_bootstrap.cc:492] T 61b4b6d4ac364a14a34733b4dd715bb3 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:12.312923 1207 tablet_bootstrap.cc:492] T 8b4bc535c42e43889401f14b6d59659e P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:12.313390 1207 ts_tablet_manager.cc:1397] T 8b4bc535c42e43889401f14b6d59659e P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.040s user 0.009s sys 0.003s
I20250114 02:54:12.314957 469 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_6e68b9b0d43c417cb8591764ef6044f5" } new_table_name: "default.Table_003a5067105b43fdaba7a58fbd24bc3a" modify_external_catalogs: true
I20250114 02:54:12.315085 463 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_b0cd93e5a498450e84fb0f1ba4be4505"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:12.315538 1207 raft_consensus.cc:357] T 8b4bc535c42e43889401f14b6d59659e P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
W20250114 02:54:12.316607 463 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b0cd93e5a498450e84fb0f1ba4be4505 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:12.298153 1394 ts_tablet_manager.cc:1905] T a4dd92b0133e413d8b895a0f33e2faf1 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:12.306308 1205 raft_consensus.cc:357] T 0752ab4b7031402db4e6be8b61cf364f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:12.320166 1205 raft_consensus.cc:383] T 0752ab4b7031402db4e6be8b61cf364f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:12.316149 1207 raft_consensus.cc:383] T 8b4bc535c42e43889401f14b6d59659e P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:12.323544 1207 raft_consensus.cc:738] T 8b4bc535c42e43889401f14b6d59659e P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:12.324066 1207 consensus_queue.cc:260] T 8b4bc535c42e43889401f14b6d59659e P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:12.327615 1205 raft_consensus.cc:738] T 0752ab4b7031402db4e6be8b61cf364f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:12.328239 468 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_acca64d3328d47578d6d1ebd594babbd" } modify_external_catalogs: true
I20250114 02:54:12.328750 468 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_acca64d3328d47578d6d1ebd594babbd" } modify_external_catalogs: true
I20250114 02:54:12.328351 1205 consensus_queue.cc:260] T 0752ab4b7031402db4e6be8b61cf364f P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:12.330565 1205 ts_tablet_manager.cc:1428] T 0752ab4b7031402db4e6be8b61cf364f P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.029s user 0.005s sys 0.000s
I20250114 02:54:12.332988 471 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_1034610abe1c4464ac9f9ebfd932ef09" } new_table_name: "default.Table_2449658d21e0468f84368f26a2756c00" modify_external_catalogs: true
I20250114 02:54:12.336998 467 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_eb7878bdce82445c97f6ce665eb52d8e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:12.338413 1206 tablet_bootstrap.cc:654] T 61b4b6d4ac364a14a34733b4dd715bb3 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
W20250114 02:54:12.338454 467 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eb7878bdce82445c97f6ce665eb52d8e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:12.339150 1207 ts_tablet_manager.cc:1428] T 8b4bc535c42e43889401f14b6d59659e P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.026s user 0.004s sys 0.000s
I20250114 02:54:12.347148 1207 tablet_bootstrap.cc:492] T 4bb9b429af92446bbe13894d5dfe4c47 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:12.356923 1205 tablet_bootstrap.cc:492] T 61b4b6d4ac364a14a34733b4dd715bb3 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:12.357424 1207 tablet_bootstrap.cc:654] T 4bb9b429af92446bbe13894d5dfe4c47 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:12.362555 1205 tablet_bootstrap.cc:654] T 61b4b6d4ac364a14a34733b4dd715bb3 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:12.373309 470 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_150e45eea89e48369a240547174217ef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:12.374776 470 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_150e45eea89e48369a240547174217ef in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:12.375427 464 catalog_manager.cc:5526] T 857119ca5f9d4ad98f899df22f32244b P 819f9e3412e54d44abb11d8b3ca877c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65). New cstate: current_term: 1 leader_uuid: "819f9e3412e54d44abb11d8b3ca877c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:12.377800 1207 tablet_bootstrap.cc:492] T 4bb9b429af92446bbe13894d5dfe4c47 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:12.378161 1207 ts_tablet_manager.cc:1397] T 4bb9b429af92446bbe13894d5dfe4c47 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.031s user 0.008s sys 0.000s
I20250114 02:54:12.380445 1206 tablet_bootstrap.cc:492] T 61b4b6d4ac364a14a34733b4dd715bb3 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:12.380355 1207 raft_consensus.cc:357] T 4bb9b429af92446bbe13894d5dfe4c47 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:12.380865 1206 ts_tablet_manager.cc:1397] T 61b4b6d4ac364a14a34733b4dd715bb3 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.072s user 0.009s sys 0.001s
I20250114 02:54:12.380703 1084 tablet_service.cc:1467] Processing CreateTablet for tablet 96b06fe4173343dabd726d253b9f9b5f (DEFAULT_TABLE table=default.Table_d1de82caa5b94791b6c7dfed1c376e8a [id=03c775a54e9e443cbc4fab853bd6d49c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:12.380960 1207 raft_consensus.cc:383] T 4bb9b429af92446bbe13894d5dfe4c47 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:12.381207 1207 raft_consensus.cc:738] T 4bb9b429af92446bbe13894d5dfe4c47 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:12.381778 1084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96b06fe4173343dabd726d253b9f9b5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:12.382180 1207 consensus_queue.cc:260] T 4bb9b429af92446bbe13894d5dfe4c47 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:12.383502 1206 raft_consensus.cc:357] T 61b4b6d4ac364a14a34733b4dd715bb3 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:12.384779 1206 raft_consensus.cc:383] T 61b4b6d4ac364a14a34733b4dd715bb3 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:12.385053 1206 raft_consensus.cc:738] T 61b4b6d4ac364a14a34733b4dd715bb3 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:12.386085 1393 ts_tablet_manager.cc:1918] T 96420dece19549b3876bd795cbdc0f0a P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:12.386644 1206 consensus_queue.cc:260] T 61b4b6d4ac364a14a34733b4dd715bb3 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:12.387436 1393 log.cc:1198] T 96420dece19549b3876bd795cbdc0f0a P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/96420dece19549b3876bd795cbdc0f0a
I20250114 02:54:12.388949 1393 ts_tablet_manager.cc:1939] T 96420dece19549b3876bd795cbdc0f0a P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:12.389902 1206 ts_tablet_manager.cc:1428] T 61b4b6d4ac364a14a34733b4dd715bb3 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.009s user 0.004s sys 0.001s
I20250114 02:54:12.392200 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet 96420dece19549b3876bd795cbdc0f0a (table default.Table_d12c2d9a4b3f42d4839548ca62645693 [id=ceeff15df8714e78b662b1cb4e4fc105]) successfully deleted
I20250114 02:54:12.393456 1206 tablet_bootstrap.cc:492] T 0752ab4b7031402db4e6be8b61cf364f P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:12.394402 796 tablet_service.cc:1467] Processing CreateTablet for tablet 96b06fe4173343dabd726d253b9f9b5f (DEFAULT_TABLE table=default.Table_d1de82caa5b94791b6c7dfed1c376e8a [id=03c775a54e9e443cbc4fab853bd6d49c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:12.398530 940 tablet_service.cc:1467] Processing CreateTablet for tablet 96b06fe4173343dabd726d253b9f9b5f (DEFAULT_TABLE table=default.Table_d1de82caa5b94791b6c7dfed1c376e8a [id=03c775a54e9e443cbc4fab853bd6d49c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:12.399924 940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96b06fe4173343dabd726d253b9f9b5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:12.402688 1207 ts_tablet_manager.cc:1428] T 4bb9b429af92446bbe13894d5dfe4c47 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.024s user 0.006s sys 0.000s
I20250114 02:54:12.404341 1207 tablet_bootstrap.cc:492] T b3b1ea0a825c4d1f8e3edf24cc74d8f7 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:12.405412 1376 raft_consensus.cc:491] T 92b8d48880434ab99eec7f1e452931fa P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:12.405815 1376 raft_consensus.cc:513] T 92b8d48880434ab99eec7f1e452931fa P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:12.407573 1206 tablet_bootstrap.cc:654] T 0752ab4b7031402db4e6be8b61cf364f P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:12.407718 1376 leader_election.cc:290] T 92b8d48880434ab99eec7f1e452931fa P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:12.409212 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92b8d48880434ab99eec7f1e452931fa" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:12.409781 818 raft_consensus.cc:2463] T 92b8d48880434ab99eec7f1e452931fa P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 0.
I20250114 02:54:12.410841 1042 leader_election.cc:304] T 92b8d48880434ab99eec7f1e452931fa P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:12.414590 941 tablet_service.cc:1467] Processing CreateTablet for tablet a3fdab330d574c0283ac58bc9e450b87 (DEFAULT_TABLE table=default.Table_67fb7ad07b314b26b85dd43b95ea1faa [id=b6874c22fe0d4151a7c8ea061f685938]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:12.415511 797 tablet_service.cc:1467] Processing CreateTablet for tablet a3fdab330d574c0283ac58bc9e450b87 (DEFAULT_TABLE table=default.Table_67fb7ad07b314b26b85dd43b95ea1faa [id=b6874c22fe0d4151a7c8ea061f685938]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:12.415807 941 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3fdab330d574c0283ac58bc9e450b87. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:12.417337 1376 raft_consensus.cc:2798] T 92b8d48880434ab99eec7f1e452931fa P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:12.417116 1086 tablet_service.cc:1467] Processing CreateTablet for tablet a3fdab330d574c0283ac58bc9e450b87 (DEFAULT_TABLE table=default.Table_67fb7ad07b314b26b85dd43b95ea1faa [id=b6874c22fe0d4151a7c8ea061f685938]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:12.417672 1376 raft_consensus.cc:491] T 92b8d48880434ab99eec7f1e452931fa P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:12.417981 1376 raft_consensus.cc:3054] T 92b8d48880434ab99eec7f1e452931fa P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:12.418318 1086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3fdab330d574c0283ac58bc9e450b87. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:12.420703 961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92b8d48880434ab99eec7f1e452931fa" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:12.421463 961 raft_consensus.cc:2463] T 92b8d48880434ab99eec7f1e452931fa P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 0.
W20250114 02:54:12.428313 455 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.125:46837, user_credentials={real_user=slave}} blocked reactor thread for 76390us
W20250114 02:54:12.428957 455 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet b63dd90f60a04465b948171e74888e79 (table default.Table_1097f09fb10e441fb639f8d7aa12d487 [id=d8c9691818d042d7b49475fa009c4ca1], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet b63dd90f60a04465b948171e74888e79 consensus state has no leader
I20250114 02:54:12.432735 1207 tablet_bootstrap.cc:654] T b3b1ea0a825c4d1f8e3edf24cc74d8f7 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:12.442446 793 tablet_service.cc:1467] Processing CreateTablet for tablet 290ad07ab8be4791a39194909e000843 (DEFAULT_TABLE table=default.Table_02b3fc3bfe484e3a9c5d5e9114deb368 [id=29d8f8e2417d403aa619839418c94207]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:12.442965 1376 raft_consensus.cc:513] T 92b8d48880434ab99eec7f1e452931fa P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:12.445456 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92b8d48880434ab99eec7f1e452931fa" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:12.445966 818 raft_consensus.cc:3054] T 92b8d48880434ab99eec7f1e452931fa P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
W20250114 02:54:12.447078 455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.107.124:39105: connect: Connection refused (error 111) [suppressed 8 similar messages]
I20250114 02:54:12.452105 818 raft_consensus.cc:2463] T 92b8d48880434ab99eec7f1e452931fa P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1.
I20250114 02:54:12.453328 1042 leader_election.cc:304] T 92b8d48880434ab99eec7f1e452931fa P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:12.454082 1309 raft_consensus.cc:2798] T 92b8d48880434ab99eec7f1e452931fa P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:12.454473 1309 raft_consensus.cc:695] T 92b8d48880434ab99eec7f1e452931fa P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Becoming Leader. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Running, Role: LEADER
I20250114 02:54:12.455159 1309 consensus_queue.cc:237] T 92b8d48880434ab99eec7f1e452931fa P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:12.462916 1376 leader_election.cc:290] T 92b8d48880434ab99eec7f1e452931fa P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Requested vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:12.463719 1235 raft_consensus.cc:491] T 0752ab4b7031402db4e6be8b61cf364f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:12.464146 1235 raft_consensus.cc:513] T 0752ab4b7031402db4e6be8b61cf364f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:12.464910 961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92b8d48880434ab99eec7f1e452931fa" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
I20250114 02:54:12.465430 961 raft_consensus.cc:3054] T 92b8d48880434ab99eec7f1e452931fa P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:12.465910 1235 leader_election.cc:290] T 0752ab4b7031402db4e6be8b61cf364f P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:12.467900 1104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0752ab4b7031402db4e6be8b61cf364f" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:12.467948 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0752ab4b7031402db4e6be8b61cf364f" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:12.468539 1104 raft_consensus.cc:2463] T 0752ab4b7031402db4e6be8b61cf364f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:12.468735 796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96b06fe4173343dabd726d253b9f9b5f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250114 02:54:12.469106 898 leader_election.cc:343] T 0752ab4b7031402db4e6be8b61cf364f P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:12.469851 793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 290ad07ab8be4791a39194909e000843. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:12.469779 898 leader_election.cc:304] T 0752ab4b7031402db4e6be8b61cf364f P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters: 819f9e3412e54d44abb11d8b3ca877c2
I20250114 02:54:12.470736 1235 raft_consensus.cc:2798] T 0752ab4b7031402db4e6be8b61cf364f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:12.471058 1235 raft_consensus.cc:491] T 0752ab4b7031402db4e6be8b61cf364f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:12.471351 1235 raft_consensus.cc:3054] T 0752ab4b7031402db4e6be8b61cf364f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:12.472498 797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3fdab330d574c0283ac58bc9e450b87. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:12.477830 1235 raft_consensus.cc:513] T 0752ab4b7031402db4e6be8b61cf364f P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:12.479800 961 raft_consensus.cc:2463] T 92b8d48880434ab99eec7f1e452931fa P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1.
I20250114 02:54:12.480414 1104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0752ab4b7031402db4e6be8b61cf364f" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:12.480950 1104 raft_consensus.cc:3054] T 0752ab4b7031402db4e6be8b61cf364f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:12.486935 1104 raft_consensus.cc:2463] T 0752ab4b7031402db4e6be8b61cf364f P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:12.488026 898 leader_election.cc:304] T 0752ab4b7031402db4e6be8b61cf364f P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:12.492668 1394 ts_tablet_manager.cc:1918] T a4dd92b0133e413d8b895a0f33e2faf1 P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:12.493065 1394 log.cc:1198] T a4dd92b0133e413d8b895a0f33e2faf1 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/a4dd92b0133e413d8b895a0f33e2faf1
I20250114 02:54:12.493748 1235 leader_election.cc:290] T 0752ab4b7031402db4e6be8b61cf364f P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Requested vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:12.494216 1235 raft_consensus.cc:2798] T 0752ab4b7031402db4e6be8b61cf364f P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:12.494597 1235 raft_consensus.cc:695] T 0752ab4b7031402db4e6be8b61cf364f P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Becoming Leader. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Running, Role: LEADER
I20250114 02:54:12.493854 1205 tablet_bootstrap.cc:492] T 61b4b6d4ac364a14a34733b4dd715bb3 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:12.495347 1205 ts_tablet_manager.cc:1397] T 61b4b6d4ac364a14a34733b4dd715bb3 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.139s user 0.018s sys 0.020s
I20250114 02:54:12.497893 940 tablet_service.cc:1467] Processing CreateTablet for tablet 290ad07ab8be4791a39194909e000843 (DEFAULT_TABLE table=default.Table_02b3fc3bfe484e3a9c5d5e9114deb368 [id=29d8f8e2417d403aa619839418c94207]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:12.499126 1084 tablet_service.cc:1467] Processing CreateTablet for tablet 290ad07ab8be4791a39194909e000843 (DEFAULT_TABLE table=default.Table_02b3fc3bfe484e3a9c5d5e9114deb368 [id=29d8f8e2417d403aa619839418c94207]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:12.499991 1084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 290ad07ab8be4791a39194909e000843. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:12.495801 1235 consensus_queue.cc:237] T 0752ab4b7031402db4e6be8b61cf364f P 525198593c8e47afaee54e74267b87e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:12.502542 936 tablet_service.cc:1467] Processing CreateTablet for tablet e3af8f1299f24b08b2d25811942aaecc (DEFAULT_TABLE table=default.Table_6e385f4df79b47cb9f1ada084a60abbf [id=be6d575261b944d6bc12cc5db002c77c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:12.503517 936 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3af8f1299f24b08b2d25811942aaecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:12.509634 940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 290ad07ab8be4791a39194909e000843. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:12.509724 1394 ts_tablet_manager.cc:1939] T a4dd92b0133e413d8b895a0f33e2faf1 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:12.509514 794 tablet_service.cc:1467] Processing CreateTablet for tablet e3af8f1299f24b08b2d25811942aaecc (DEFAULT_TABLE table=default.Table_6e385f4df79b47cb9f1ada084a60abbf [id=be6d575261b944d6bc12cc5db002c77c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:12.510511 794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3af8f1299f24b08b2d25811942aaecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:12.512116 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet a4dd92b0133e413d8b895a0f33e2faf1 (table default.Table_075cadc0bfa145718aa9f2f6850f595c [id=ec0fc676edc842b2b7114a4b3fe0626a]) successfully deleted
I20250114 02:54:12.512913 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0752ab4b7031402db4e6be8b61cf364f" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
W20250114 02:54:12.514050 898 leader_election.cc:343] T 0752ab4b7031402db4e6be8b61cf364f P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:12.521819 1205 raft_consensus.cc:357] T 61b4b6d4ac364a14a34733b4dd715bb3 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:12.522435 1205 raft_consensus.cc:383] T 61b4b6d4ac364a14a34733b4dd715bb3 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:12.522655 1205 raft_consensus.cc:738] T 61b4b6d4ac364a14a34733b4dd715bb3 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:12.523172 1205 consensus_queue.cc:260] T 61b4b6d4ac364a14a34733b4dd715bb3 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
W20250114 02:54:12.526314 455 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.124:39105, user_credentials={real_user=slave}} blocked reactor thread for 79248us
I20250114 02:54:12.530033 1207 tablet_bootstrap.cc:492] T b3b1ea0a825c4d1f8e3edf24cc74d8f7 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:12.530462 1207 ts_tablet_manager.cc:1397] T b3b1ea0a825c4d1f8e3edf24cc74d8f7 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.127s user 0.016s sys 0.002s
I20250114 02:54:12.533533 1207 raft_consensus.cc:357] T b3b1ea0a825c4d1f8e3edf24cc74d8f7 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:12.540938 937 tablet_service.cc:1467] Processing CreateTablet for tablet 15704785f56e43a3af69f181c75877ce (DEFAULT_TABLE table=default.Table_95f4830fe0bc41519e53f94f9514583a [id=1f964e9ed159449580fe5326673c18ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:12.541950 937 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15704785f56e43a3af69f181c75877ce. 1 dirs total, 0 dirs full, 0 dirs failed
W20250114 02:54:12.543514 471 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet de4ca83a69704a368a25954cdc1b9392 has no consensus state
I20250114 02:54:12.546615 794 tablet_service.cc:1467] Processing CreateTablet for tablet 15704785f56e43a3af69f181c75877ce (DEFAULT_TABLE table=default.Table_95f4830fe0bc41519e53f94f9514583a [id=1f964e9ed159449580fe5326673c18ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:12.540004 1081 tablet_service.cc:1467] Processing CreateTablet for tablet e3af8f1299f24b08b2d25811942aaecc (DEFAULT_TABLE table=default.Table_6e385f4df79b47cb9f1ada084a60abbf [id=be6d575261b944d6bc12cc5db002c77c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:12.547825 1207 raft_consensus.cc:383] T b3b1ea0a825c4d1f8e3edf24cc74d8f7 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:12.548108 1207 raft_consensus.cc:738] T b3b1ea0a825c4d1f8e3edf24cc74d8f7 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:12.548592 1081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3af8f1299f24b08b2d25811942aaecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:12.550164 1206 tablet_bootstrap.cc:492] T 0752ab4b7031402db4e6be8b61cf364f P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:12.550529 1206 ts_tablet_manager.cc:1397] T 0752ab4b7031402db4e6be8b61cf364f P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.157s user 0.018s sys 0.010s
I20250114 02:54:12.544461 1085 tablet_service.cc:1467] Processing CreateTablet for tablet 15704785f56e43a3af69f181c75877ce (DEFAULT_TABLE table=default.Table_95f4830fe0bc41519e53f94f9514583a [id=1f964e9ed159449580fe5326673c18ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:12.552839 1085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15704785f56e43a3af69f181c75877ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:12.552668 1206 raft_consensus.cc:357] T 0752ab4b7031402db4e6be8b61cf364f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:12.553287 1206 raft_consensus.cc:383] T 0752ab4b7031402db4e6be8b61cf364f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:12.553553 1206 raft_consensus.cc:738] T 0752ab4b7031402db4e6be8b61cf364f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:12.554216 1207 consensus_queue.cc:260] T b3b1ea0a825c4d1f8e3edf24cc74d8f7 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:12.555810 1206 consensus_queue.cc:260] T 0752ab4b7031402db4e6be8b61cf364f P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:12.556735 468 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet 83b5b36a26e74668998a19fa0ddf7188
I20250114 02:54:12.557755 463 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_bacdf22532c84100a69a38cfa0ad07e5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:12.559275 463 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bacdf22532c84100a69a38cfa0ad07e5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:12.560134 941 tablet_service.cc:1514] Processing DeleteTablet for tablet 83b5b36a26e74668998a19fa0ddf7188 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:12 UTC) from {username='slave'} at 127.0.0.1:55068
W20250114 02:54:12.561700 456 catalog_manager.cc:4852] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet 83b5b36a26e74668998a19fa0ddf7188 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 83b5b36a26e74668998a19fa0ddf7188 already in progress: creating tablet
I20250114 02:54:12.565820 793 tablet_service.cc:1514] Processing DeleteTablet for tablet 83b5b36a26e74668998a19fa0ddf7188 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:12 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:12.567201 1207 ts_tablet_manager.cc:1428] T b3b1ea0a825c4d1f8e3edf24cc74d8f7 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.036s user 0.005s sys 0.000s
I20250114 02:54:12.568008 1207 tablet_bootstrap.cc:492] T 276942e5e54246c79f0d4f08297f58ea P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:12.572335 1205 ts_tablet_manager.cc:1428] T 61b4b6d4ac364a14a34733b4dd715bb3 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.077s user 0.006s sys 0.000s
I20250114 02:54:12.572508 1294 raft_consensus.cc:491] T 0752ab4b7031402db4e6be8b61cf364f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:12.572934 1294 raft_consensus.cc:513] T 0752ab4b7031402db4e6be8b61cf364f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:12.573664 1206 ts_tablet_manager.cc:1428] T 0752ab4b7031402db4e6be8b61cf364f P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.023s user 0.003s sys 0.003s
I20250114 02:54:12.574642 1206 tablet_bootstrap.cc:492] T 4bb9b429af92446bbe13894d5dfe4c47 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
W20250114 02:54:12.574836 458 catalog_manager.cc:4852] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet 83b5b36a26e74668998a19fa0ddf7188 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 83b5b36a26e74668998a19fa0ddf7188 already in progress: creating tablet
I20250114 02:54:12.574708 1294 leader_election.cc:290] T 0752ab4b7031402db4e6be8b61cf364f P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:12.579021 1104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0752ab4b7031402db4e6be8b61cf364f" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:12.579733 1104 raft_consensus.cc:2388] T 0752ab4b7031402db4e6be8b61cf364f P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 819f9e3412e54d44abb11d8b3ca877c2 in current term 1: Already voted for candidate 525198593c8e47afaee54e74267b87e3 in this term.
I20250114 02:54:12.576421 1207 tablet_bootstrap.cc:654] T 276942e5e54246c79f0d4f08297f58ea P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:12.583647 961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0752ab4b7031402db4e6be8b61cf364f" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:12.585326 751 leader_election.cc:304] T 0752ab4b7031402db4e6be8b61cf364f P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:12.586423 1294 raft_consensus.cc:3054] T 0752ab4b7031402db4e6be8b61cf364f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:12.588436 1205 tablet_bootstrap.cc:492] T b3b1ea0a825c4d1f8e3edf24cc74d8f7 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:12.594291 1294 raft_consensus.cc:2743] T 0752ab4b7031402db4e6be8b61cf364f P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:12.599207 1309 raft_consensus.cc:491] T 8b4bc535c42e43889401f14b6d59659e P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:12.599612 1309 raft_consensus.cc:513] T 8b4bc535c42e43889401f14b6d59659e P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:12.603515 1309 leader_election.cc:290] T 8b4bc535c42e43889401f14b6d59659e P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:12.605886 1205 tablet_bootstrap.cc:654] T b3b1ea0a825c4d1f8e3edf24cc74d8f7 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:12.605664 961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b4bc535c42e43889401f14b6d59659e" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
W20250114 02:54:12.607173 1040 leader_election.cc:343] T 8b4bc535c42e43889401f14b6d59659e P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:12.602497 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b4bc535c42e43889401f14b6d59659e" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:12.608469 1086 tablet_service.cc:1514] Processing DeleteTablet for tablet 83b5b36a26e74668998a19fa0ddf7188 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:12 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:12.604346 794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15704785f56e43a3af69f181c75877ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:12.609479 1389 raft_consensus.cc:491] T b4740f6b3a6741368ba68907ad065279 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 02:54:12.610010 458 catalog_manager.cc:4852] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 83b5b36a26e74668998a19fa0ddf7188 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 83b5b36a26e74668998a19fa0ddf7188 already in progress: creating tablet
I20250114 02:54:12.609880 1389 raft_consensus.cc:513] T b4740f6b3a6741368ba68907ad065279 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:12.611774 1389 leader_election.cc:290] T b4740f6b3a6741368ba68907ad065279 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
W20250114 02:54:12.612373 1042 leader_election.cc:343] T 8b4bc535c42e43889401f14b6d59659e P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:12.612529 1104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4740f6b3a6741368ba68907ad065279" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:12.612772 1042 leader_election.cc:304] T 8b4bc535c42e43889401f14b6d59659e P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90d6c2c58bb54e0eafb365625ac33871; no voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2
I20250114 02:54:12.613180 1104 raft_consensus.cc:2463] T b4740f6b3a6741368ba68907ad065279 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 0.
I20250114 02:54:12.614464 1309 raft_consensus.cc:2743] T 8b4bc535c42e43889401f14b6d59659e P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:12.615777 753 leader_election.cc:304] T b4740f6b3a6741368ba68907ad065279 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:12.617030 1389 raft_consensus.cc:2798] T b4740f6b3a6741368ba68907ad065279 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:12.617369 1389 raft_consensus.cc:491] T b4740f6b3a6741368ba68907ad065279 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:12.617691 1389 raft_consensus.cc:3054] T b4740f6b3a6741368ba68907ad065279 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:12.626662 1389 raft_consensus.cc:513] T b4740f6b3a6741368ba68907ad065279 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:12.627887 961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4740f6b3a6741368ba68907ad065279" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:12.628715 961 raft_consensus.cc:2463] T b4740f6b3a6741368ba68907ad065279 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 0.
I20250114 02:54:12.641417 1205 tablet_bootstrap.cc:492] T b3b1ea0a825c4d1f8e3edf24cc74d8f7 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:12.641780 1205 ts_tablet_manager.cc:1397] T b3b1ea0a825c4d1f8e3edf24cc74d8f7 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.054s user 0.011s sys 0.000s
I20250114 02:54:12.644294 1205 raft_consensus.cc:357] T b3b1ea0a825c4d1f8e3edf24cc74d8f7 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:12.644907 1205 raft_consensus.cc:383] T b3b1ea0a825c4d1f8e3edf24cc74d8f7 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:12.645128 1205 raft_consensus.cc:738] T b3b1ea0a825c4d1f8e3edf24cc74d8f7 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:12.645680 1205 consensus_queue.cc:260] T b3b1ea0a825c4d1f8e3edf24cc74d8f7 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:12.656869 1205 ts_tablet_manager.cc:1428] T b3b1ea0a825c4d1f8e3edf24cc74d8f7 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20250114 02:54:12.657684 1205 tablet_bootstrap.cc:492] T 8b4bc535c42e43889401f14b6d59659e P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:12.666188 1309 raft_consensus.cc:491] T 4bb9b429af92446bbe13894d5dfe4c47 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:12.666647 1309 raft_consensus.cc:513] T 4bb9b429af92446bbe13894d5dfe4c47 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:12.668445 1309 leader_election.cc:290] T 4bb9b429af92446bbe13894d5dfe4c47 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:12.673410 1205 tablet_bootstrap.cc:654] T 8b4bc535c42e43889401f14b6d59659e P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:12.678915 961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bb9b429af92446bbe13894d5dfe4c47" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
W20250114 02:54:12.680330 1040 leader_election.cc:343] T 4bb9b429af92446bbe13894d5dfe4c47 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:12.639851 1206 tablet_bootstrap.cc:654] T 4bb9b429af92446bbe13894d5dfe4c47 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:12.690018 466 catalog_manager.cc:5526] T d434cbc9daf549a6bdc04c8c76ead8a6 P 90d6c2c58bb54e0eafb365625ac33871 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67). New cstate: current_term: 1 leader_uuid: "90d6c2c58bb54e0eafb365625ac33871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:12.669816 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bb9b429af92446bbe13894d5dfe4c47" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:12.690956 1104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4740f6b3a6741368ba68907ad065279" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:12.684149 1385 consensus_queue.cc:1035] T a3886c5f976445cebc95ef1839a74aa9 P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:12.691491 1104 raft_consensus.cc:3054] T b4740f6b3a6741368ba68907ad065279 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
W20250114 02:54:12.692770 1042 leader_election.cc:343] T 4bb9b429af92446bbe13894d5dfe4c47 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:12.693156 1042 leader_election.cc:304] T 4bb9b429af92446bbe13894d5dfe4c47 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90d6c2c58bb54e0eafb365625ac33871; no voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2
I20250114 02:54:12.694064 1309 raft_consensus.cc:2743] T 4bb9b429af92446bbe13894d5dfe4c47 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250114 02:54:12.694597 469 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d434cbc9daf549a6bdc04c8c76ead8a6 consensus state has no leader
I20250114 02:54:12.699100 1104 raft_consensus.cc:2463] T b4740f6b3a6741368ba68907ad065279 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 1.
I20250114 02:54:12.700711 1385 raft_consensus.cc:491] T 61b4b6d4ac364a14a34733b4dd715bb3 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:12.701104 753 leader_election.cc:304] T b4740f6b3a6741368ba68907ad065279 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:12.701087 1385 raft_consensus.cc:513] T 61b4b6d4ac364a14a34733b4dd715bb3 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:12.701978 1294 raft_consensus.cc:2798] T b4740f6b3a6741368ba68907ad065279 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:12.702854 1385 leader_election.cc:290] T 61b4b6d4ac364a14a34733b4dd715bb3 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:12.703428 1105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b4b6d4ac364a14a34733b4dd715bb3" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:12.703670 1389 leader_election.cc:290] T b4740f6b3a6741368ba68907ad065279 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Requested vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:12.704003 1105 raft_consensus.cc:2463] T 61b4b6d4ac364a14a34733b4dd715bb3 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 0.
I20250114 02:54:12.704982 753 leader_election.cc:304] T 61b4b6d4ac364a14a34733b4dd715bb3 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:12.706009 961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b4b6d4ac364a14a34733b4dd715bb3" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:12.706880 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4740f6b3a6741368ba68907ad065279" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
I20250114 02:54:12.707428 962 raft_consensus.cc:3054] T b4740f6b3a6741368ba68907ad065279 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:12.707464 961 raft_consensus.cc:2463] T 61b4b6d4ac364a14a34733b4dd715bb3 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 0.
I20250114 02:54:12.709923 1389 raft_consensus.cc:2798] T 61b4b6d4ac364a14a34733b4dd715bb3 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:12.710328 1389 raft_consensus.cc:491] T 61b4b6d4ac364a14a34733b4dd715bb3 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:12.710675 1389 raft_consensus.cc:3054] T 61b4b6d4ac364a14a34733b4dd715bb3 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:12.714919 962 raft_consensus.cc:2463] T b4740f6b3a6741368ba68907ad065279 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 1.
I20250114 02:54:12.715555 1206 tablet_bootstrap.cc:492] T 4bb9b429af92446bbe13894d5dfe4c47 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:12.715931 1206 ts_tablet_manager.cc:1397] T 4bb9b429af92446bbe13894d5dfe4c47 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.141s user 0.019s sys 0.005s
I20250114 02:54:12.718048 1206 raft_consensus.cc:357] T 4bb9b429af92446bbe13894d5dfe4c47 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:12.718634 1206 raft_consensus.cc:383] T 4bb9b429af92446bbe13894d5dfe4c47 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:12.718843 1206 raft_consensus.cc:738] T 4bb9b429af92446bbe13894d5dfe4c47 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:12.719302 1206 consensus_queue.cc:260] T 4bb9b429af92446bbe13894d5dfe4c47 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:12.724594 1294 raft_consensus.cc:695] T b4740f6b3a6741368ba68907ad065279 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 LEADER]: Becoming Leader. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Running, Role: LEADER
W20250114 02:54:12.725818 455 consensus_peers.cc:487] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750 -> Peer 87b3db09ac87436e9a60d6de5fedaff5 (127.0.107.124:39105): Couldn't send request to peer 87b3db09ac87436e9a60d6de5fedaff5. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.124:39105: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250114 02:54:12.725561 1294 consensus_queue.cc:237] T b4740f6b3a6741368ba68907ad065279 P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:12.734319 1205 tablet_bootstrap.cc:492] T 8b4bc535c42e43889401f14b6d59659e P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:12.734690 1205 ts_tablet_manager.cc:1397] T 8b4bc535c42e43889401f14b6d59659e P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.077s user 0.003s sys 0.006s
I20250114 02:54:12.736851 1205 raft_consensus.cc:357] T 8b4bc535c42e43889401f14b6d59659e P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:12.737475 1205 raft_consensus.cc:383] T 8b4bc535c42e43889401f14b6d59659e P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:12.737691 1205 raft_consensus.cc:738] T 8b4bc535c42e43889401f14b6d59659e P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:12.738992 1205 consensus_queue.cc:260] T 8b4bc535c42e43889401f14b6d59659e P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:12.740834 1205 ts_tablet_manager.cc:1428] T 8b4bc535c42e43889401f14b6d59659e P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250114 02:54:12.741947 1205 tablet_bootstrap.cc:492] T b63dd90f60a04465b948171e74888e79 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:12.747308 1205 tablet_bootstrap.cc:654] T b63dd90f60a04465b948171e74888e79 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:12.749727 463 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_4da7342e02e441008994800e5215bcae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:12.751494 463 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4da7342e02e441008994800e5215bcae in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:12.755005 1235 raft_consensus.cc:491] T b3b1ea0a825c4d1f8e3edf24cc74d8f7 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:12.755402 1235 raft_consensus.cc:513] T b3b1ea0a825c4d1f8e3edf24cc74d8f7 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:12.756714 469 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_661ed49bcbbb476aa68742b72604d95f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:12.757170 1235 leader_election.cc:290] T b3b1ea0a825c4d1f8e3edf24cc74d8f7 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:12.758034 1206 ts_tablet_manager.cc:1428] T 4bb9b429af92446bbe13894d5dfe4c47 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.042s user 0.005s sys 0.001s
W20250114 02:54:12.758044 469 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_661ed49bcbbb476aa68742b72604d95f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:12.759810 1206 tablet_bootstrap.cc:492] T 276942e5e54246c79f0d4f08297f58ea P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:12.760558 470 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_893071c3e2ec472a81de4617fc929071" } new_table_name: "default.Table_0931c459ae794d2091fcc0f51d031548" modify_external_catalogs: true
I20250114 02:54:12.763283 468 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_55dad15f1b4f4e22854ba3070b0eea8e" } modify_external_catalogs: true
I20250114 02:54:12.763775 468 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_55dad15f1b4f4e22854ba3070b0eea8e" } modify_external_catalogs: true
I20250114 02:54:12.764437 1309 consensus_queue.cc:1035] T d434cbc9daf549a6bdc04c8c76ead8a6 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:12.766999 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3b1ea0a825c4d1f8e3edf24cc74d8f7" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:12.767305 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3b1ea0a825c4d1f8e3edf24cc74d8f7" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:12.767586 429 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrCUXNq/build/tsan/bin/kudu
/tmp/dist-test-taskrCUXNq/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.107.124:39105
--webserver_interface=127.0.107.124
--webserver_port=36399
--builtin_ntp_servers=127.0.107.84:40455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.107.126:45367,127.0.107.125:46837,127.0.107.124:39105
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250114 02:54:12.767889 1106 raft_consensus.cc:2463] T b3b1ea0a825c4d1f8e3edf24cc74d8f7 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:12.769258 898 leader_election.cc:304] T b3b1ea0a825c4d1f8e3edf24cc74d8f7 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters:
W20250114 02:54:12.770098 898 leader_election.cc:343] T b3b1ea0a825c4d1f8e3edf24cc74d8f7 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:12.770170 1309 consensus_queue.cc:1035] T d434cbc9daf549a6bdc04c8c76ead8a6 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:12.773252 1408 raft_consensus.cc:2798] T b3b1ea0a825c4d1f8e3edf24cc74d8f7 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:12.773598 1408 raft_consensus.cc:491] T b3b1ea0a825c4d1f8e3edf24cc74d8f7 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:12.773886 1408 raft_consensus.cc:3054] T b3b1ea0a825c4d1f8e3edf24cc74d8f7 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:12.790016 1408 raft_consensus.cc:513] T b3b1ea0a825c4d1f8e3edf24cc74d8f7 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:12.809374 471 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_f290e54e05c7429c9d991c02cfdfea48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:12.811434 817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3b1ea0a825c4d1f8e3edf24cc74d8f7" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
W20250114 02:54:12.813311 898 leader_election.cc:343] T b3b1ea0a825c4d1f8e3edf24cc74d8f7 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:12.813174 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3b1ea0a825c4d1f8e3edf24cc74d8f7" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:12.814306 1106 raft_consensus.cc:3054] T b3b1ea0a825c4d1f8e3edf24cc74d8f7 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:12.813828 1206 tablet_bootstrap.cc:654] T 276942e5e54246c79f0d4f08297f58ea P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:12.821537 1106 raft_consensus.cc:2463] T b3b1ea0a825c4d1f8e3edf24cc74d8f7 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:12.822618 898 leader_election.cc:304] T b3b1ea0a825c4d1f8e3edf24cc74d8f7 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters: 819f9e3412e54d44abb11d8b3ca877c2
I20250114 02:54:12.823417 1235 raft_consensus.cc:2798] T b3b1ea0a825c4d1f8e3edf24cc74d8f7 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election won for term 1
W20250114 02:54:12.824507 471 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f290e54e05c7429c9d991c02cfdfea48 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:12.838307 1408 leader_election.cc:290] T b3b1ea0a825c4d1f8e3edf24cc74d8f7 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Requested vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:12.847923 1421 consensus_queue.cc:1035] T a3886c5f976445cebc95ef1839a74aa9 P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:54:12.859766 1235 raft_consensus.cc:695] T b3b1ea0a825c4d1f8e3edf24cc74d8f7 P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Becoming Leader. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Running, Role: LEADER
I20250114 02:54:12.864454 1207 tablet_bootstrap.cc:492] T 276942e5e54246c79f0d4f08297f58ea P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:12.864840 1207 ts_tablet_manager.cc:1397] T 276942e5e54246c79f0d4f08297f58ea P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.297s user 0.014s sys 0.054s
I20250114 02:54:12.867411 1207 raft_consensus.cc:357] T 276942e5e54246c79f0d4f08297f58ea P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:12.868163 1207 raft_consensus.cc:383] T 276942e5e54246c79f0d4f08297f58ea P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:12.868454 1207 raft_consensus.cc:738] T 276942e5e54246c79f0d4f08297f58ea P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:12.869096 1207 consensus_queue.cc:260] T 276942e5e54246c79f0d4f08297f58ea P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:12.871632 1207 ts_tablet_manager.cc:1428] T 276942e5e54246c79f0d4f08297f58ea P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250114 02:54:12.868054 1235 consensus_queue.cc:237] T b3b1ea0a825c4d1f8e3edf24cc74d8f7 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:12.877550 1207 tablet_bootstrap.cc:492] T 83b5b36a26e74668998a19fa0ddf7188 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:12.878597 1205 tablet_bootstrap.cc:492] T b63dd90f60a04465b948171e74888e79 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:12.878989 1205 ts_tablet_manager.cc:1397] T b63dd90f60a04465b948171e74888e79 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.137s user 0.004s sys 0.006s
I20250114 02:54:12.889735 1205 raft_consensus.cc:357] T b63dd90f60a04465b948171e74888e79 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:12.890368 1205 raft_consensus.cc:383] T b63dd90f60a04465b948171e74888e79 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:12.890570 1205 raft_consensus.cc:738] T b63dd90f60a04465b948171e74888e79 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:12.898597 471 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_2a1287091fb9466d9e2fae14499a0963" } new_table_name: "default.Table_acb5bf9354be440bbf6034cf1c535dab" modify_external_catalogs: true
I20250114 02:54:12.904206 1205 consensus_queue.cc:260] T b63dd90f60a04465b948171e74888e79 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:12.906175 1205 ts_tablet_manager.cc:1428] T b63dd90f60a04465b948171e74888e79 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.027s user 0.005s sys 0.000s
I20250114 02:54:12.904021 1389 raft_consensus.cc:513] T 61b4b6d4ac364a14a34733b4dd715bb3 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:12.923005 464 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 61b4b6d4ac364a14a34733b4dd715bb3 (table default.Table_893071c3e2ec472a81de4617fc929071 [id=e279c31bd63b40b9ade03a3b928bf153]). Expected version 1 got 0
I20250114 02:54:12.927114 937 tablet_service.cc:1467] Processing CreateTablet for tablet 00080bea6c8e46e584ccba6dcb0e221b (DEFAULT_TABLE table=default.Table_eb7878bdce82445c97f6ce665eb52d8e [id=3adecb85db3b44a4a65e50e61a91e908]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:12.927632 466 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet 61b4b6d4ac364a14a34733b4dd715bb3 (table default.Table_893071c3e2ec472a81de4617fc929071 [id=e279c31bd63b40b9ade03a3b928bf153]). Expected version 1 got 0
I20250114 02:54:12.928315 937 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00080bea6c8e46e584ccba6dcb0e221b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:12.928332 466 catalog_manager.cc:5526] T 92b8d48880434ab99eec7f1e452931fa P 90d6c2c58bb54e0eafb365625ac33871 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67). New cstate: current_term: 1 leader_uuid: "90d6c2c58bb54e0eafb365625ac33871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:12.930135 466 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet a3886c5f976445cebc95ef1839a74aa9 (table default.Table_35a78676616e4b20906b262c9d0c0d9e [id=e984149e9da444f1b2802e3f35a60fd6]). Expected version 2 got 0
I20250114 02:54:12.930624 466 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet 8b4bc535c42e43889401f14b6d59659e (table default.Table_d90b2995b13b4a8fb7e50cc4b79cbf84 [id=f8e40e0aa44b4157ac3438d11685b2e7]). Expected version 1 got 0
I20250114 02:54:12.931456 462 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet 0752ab4b7031402db4e6be8b61cf364f
I20250114 02:54:12.931429 940 tablet_service.cc:1467] Processing CreateTablet for tablet 7ecf782bb3d54528aa996abca1ea3a15 (DEFAULT_TABLE table=default.Table_b0cd93e5a498450e84fb0f1ba4be4505 [id=c70751b59bfc4c6fa60d0e8bb40d5f36]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:12.932493 940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ecf782bb3d54528aa996abca1ea3a15. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:12.935520 797 tablet_service.cc:1467] Processing CreateTablet for tablet 00080bea6c8e46e584ccba6dcb0e221b (DEFAULT_TABLE table=default.Table_eb7878bdce82445c97f6ce665eb52d8e [id=3adecb85db3b44a4a65e50e61a91e908]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:12.936411 1085 tablet_service.cc:1514] Processing DeleteTablet for tablet 0752ab4b7031402db4e6be8b61cf364f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:12 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:12.936589 797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00080bea6c8e46e584ccba6dcb0e221b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:12.937121 936 tablet_service.cc:1514] Processing DeleteTablet for tablet 0752ab4b7031402db4e6be8b61cf364f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:12 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:12.937826 1223 tablet_replica.cc:331] T 0752ab4b7031402db4e6be8b61cf364f P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:12.938505 1223 raft_consensus.cc:2238] T 0752ab4b7031402db4e6be8b61cf364f P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:12.938903 1223 raft_consensus.cc:2267] T 0752ab4b7031402db4e6be8b61cf364f P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:12.940490 1393 tablet_replica.cc:331] T 0752ab4b7031402db4e6be8b61cf364f P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:12.942001 1393 raft_consensus.cc:2238] T 0752ab4b7031402db4e6be8b61cf364f P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Raft consensus shutting down.
I20250114 02:54:12.942665 1393 pending_rounds.cc:62] T 0752ab4b7031402db4e6be8b61cf364f P 525198593c8e47afaee54e74267b87e3: Trying to abort 1 pending ops.
I20250114 02:54:12.942899 1393 pending_rounds.cc:69] T 0752ab4b7031402db4e6be8b61cf364f P 525198593c8e47afaee54e74267b87e3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114028042252595200 op_type: NO_OP noop_request { }
I20250114 02:54:12.943262 1393 raft_consensus.cc:2883] T 0752ab4b7031402db4e6be8b61cf364f P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250114 02:54:12.943573 1393 raft_consensus.cc:2267] T 0752ab4b7031402db4e6be8b61cf364f P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:12.943727 1085 tablet_service.cc:1467] Processing CreateTablet for tablet 7ecf782bb3d54528aa996abca1ea3a15 (DEFAULT_TABLE table=default.Table_b0cd93e5a498450e84fb0f1ba4be4505 [id=c70751b59bfc4c6fa60d0e8bb40d5f36]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:12.944912 936 tablet_service.cc:1467] Processing CreateTablet for tablet 961b6a25a13a4fa2a732a6383dcd8c84 (DEFAULT_TABLE table=default.Table_c6a60d505816422d91eccbe4813d7843 [id=e3418787ef0a4e22979fb41d4f94ebb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:12.945886 936 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 961b6a25a13a4fa2a732a6383dcd8c84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:12.947492 468 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet 857119ca5f9d4ad98f899df22f32244b
I20250114 02:54:12.947862 793 tablet_service.cc:1514] Processing DeleteTablet for tablet 0752ab4b7031402db4e6be8b61cf364f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:12 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:12.949645 1394 tablet_replica.cc:331] T 0752ab4b7031402db4e6be8b61cf364f P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:12.950191 1086 tablet_service.cc:1514] Processing DeleteTablet for tablet 857119ca5f9d4ad98f899df22f32244b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:12 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:12.950404 1394 raft_consensus.cc:2238] T 0752ab4b7031402db4e6be8b61cf364f P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:12.950841 1394 raft_consensus.cc:2267] T 0752ab4b7031402db4e6be8b61cf364f P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:12.952131 798 tablet_service.cc:1467] Processing CreateTablet for tablet 7ecf782bb3d54528aa996abca1ea3a15 (DEFAULT_TABLE table=default.Table_b0cd93e5a498450e84fb0f1ba4be4505 [id=c70751b59bfc4c6fa60d0e8bb40d5f36]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:12.953198 798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ecf782bb3d54528aa996abca1ea3a15. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:12.953275 1085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ecf782bb3d54528aa996abca1ea3a15. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:12.956766 462 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_322900c20b6647489b2f2a4deecb7879"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:12.958410 462 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_322900c20b6647489b2f2a4deecb7879 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:12.961563 1394 ts_tablet_manager.cc:1905] T 0752ab4b7031402db4e6be8b61cf364f P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:12.962499 1086 tablet_service.cc:1467] Processing CreateTablet for tablet 961b6a25a13a4fa2a732a6383dcd8c84 (DEFAULT_TABLE table=default.Table_c6a60d505816422d91eccbe4813d7843 [id=e3418787ef0a4e22979fb41d4f94ebb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:12.962880 941 tablet_service.cc:1514] Processing DeleteTablet for tablet 857119ca5f9d4ad98f899df22f32244b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:12 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:12.963521 1086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 961b6a25a13a4fa2a732a6383dcd8c84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:12.964499 1223 ts_tablet_manager.cc:1905] T 0752ab4b7031402db4e6be8b61cf364f P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:12.970736 796 tablet_service.cc:1514] Processing DeleteTablet for tablet 857119ca5f9d4ad98f899df22f32244b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:12 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:12.982071 941 tablet_service.cc:1467] Processing CreateTablet for tablet de4ca83a69704a368a25954cdc1b9392 (DEFAULT_TABLE table=default.Table_2449658d21e0468f84368f26a2756c00 [id=8536eb7f6bf542daa88a8db0e4873456]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:12.982708 796 tablet_service.cc:1467] Processing CreateTablet for tablet 961b6a25a13a4fa2a732a6383dcd8c84 (DEFAULT_TABLE table=default.Table_c6a60d505816422d91eccbe4813d7843 [id=e3418787ef0a4e22979fb41d4f94ebb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:12.983012 941 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de4ca83a69704a368a25954cdc1b9392. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:12.983654 796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 961b6a25a13a4fa2a732a6383dcd8c84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:12.906893 1205 tablet_bootstrap.cc:492] T 4bb9b429af92446bbe13894d5dfe4c47 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:13.000308 1081 tablet_service.cc:1467] Processing CreateTablet for tablet 00080bea6c8e46e584ccba6dcb0e221b (DEFAULT_TABLE table=default.Table_eb7878bdce82445c97f6ce665eb52d8e [id=3adecb85db3b44a4a65e50e61a91e908]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:13.001528 1081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00080bea6c8e46e584ccba6dcb0e221b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:13.002338 1084 tablet_service.cc:1467] Processing CreateTablet for tablet de4ca83a69704a368a25954cdc1b9392 (DEFAULT_TABLE table=default.Table_2449658d21e0468f84368f26a2756c00 [id=8536eb7f6bf542daa88a8db0e4873456]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:13.003185 1084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de4ca83a69704a368a25954cdc1b9392. 1 dirs total, 0 dirs full, 0 dirs failed
W20250114 02:54:13.009779 471 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4bb9b429af92446bbe13894d5dfe4c47 consensus state has no leader
W20250114 02:54:13.010859 470 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 61b4b6d4ac364a14a34733b4dd715bb3 consensus state has no leader
I20250114 02:54:13.021220 464 master_service.cc:946] ReplaceTablet: received request to replace tablet c20aca761313498193e1d79c10705f8f from {username='slave'} at 127.0.0.1:33540
I20250114 02:54:13.030234 1415 consensus_queue.cc:1035] T 857119ca5f9d4ad98f899df22f32244b P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:54:13.037869 1393 ts_tablet_manager.cc:1905] T 0752ab4b7031402db4e6be8b61cf364f P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:13.041380 793 tablet_service.cc:1467] Processing CreateTablet for tablet de4ca83a69704a368a25954cdc1b9392 (DEFAULT_TABLE table=default.Table_2449658d21e0468f84368f26a2756c00 [id=8536eb7f6bf542daa88a8db0e4873456]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:13.042426 793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de4ca83a69704a368a25954cdc1b9392. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:13.043394 1083 tablet_service.cc:1467] Processing CreateTablet for tablet db24bf3db61941ba9b5e0662db23e748 (DEFAULT_TABLE table=default.Table_494541d472dc48d0bef33a634c1180fa [id=5ae72503278349c0bbe315174c896074]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:13.044624 1083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db24bf3db61941ba9b5e0662db23e748. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:13.045079 792 tablet_service.cc:1467] Processing CreateTablet for tablet db24bf3db61941ba9b5e0662db23e748 (DEFAULT_TABLE table=default.Table_494541d472dc48d0bef33a634c1180fa [id=5ae72503278349c0bbe315174c896074]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:13.046025 792 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db24bf3db61941ba9b5e0662db23e748. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:13.049227 939 tablet_service.cc:1467] Processing CreateTablet for tablet db24bf3db61941ba9b5e0662db23e748 (DEFAULT_TABLE table=default.Table_494541d472dc48d0bef33a634c1180fa [id=5ae72503278349c0bbe315174c896074]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:13.050258 939 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db24bf3db61941ba9b5e0662db23e748. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:13.053076 470 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_393cdd3f26b649d6b7b51051237e2c81"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:13.054497 470 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_393cdd3f26b649d6b7b51051237e2c81 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:13.056118 471 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_efc3b170c00f4ce09b2601f7f5ebf3b8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:13.057643 471 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_efc3b170c00f4ce09b2601f7f5ebf3b8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:13.072654 1205 tablet_bootstrap.cc:654] T 4bb9b429af92446bbe13894d5dfe4c47 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:13.072772 940 tablet_service.cc:1467] Processing CreateTablet for tablet 9812754c610548d1b43084bbc94fc3e4 (DEFAULT_TABLE table=default.Table_2217d3b1dcf4455d9103cad045518573 [id=2199e16ab337450ebc7c664318521edf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:13.083309 1086 tablet_service.cc:1467] Processing CreateTablet for tablet 9812754c610548d1b43084bbc94fc3e4 (DEFAULT_TABLE table=default.Table_2217d3b1dcf4455d9103cad045518573 [id=2199e16ab337450ebc7c664318521edf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:13.089527 468 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_57e1a7bbbda14363912be7c5f5cdb508"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:13.091153 468 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_57e1a7bbbda14363912be7c5f5cdb508 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:13.101473 940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9812754c610548d1b43084bbc94fc3e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:13.100831 1086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9812754c610548d1b43084bbc94fc3e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:13.116873 1433 tablet.cc:1721] T a3886c5f976445cebc95ef1839a74aa9 P 819f9e3412e54d44abb11d8b3ca877c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250114 02:54:13.120999 798 tablet_service.cc:1467] Processing CreateTablet for tablet 9812754c610548d1b43084bbc94fc3e4 (DEFAULT_TABLE table=default.Table_2217d3b1dcf4455d9103cad045518573 [id=2199e16ab337450ebc7c664318521edf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:13.122543 798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9812754c610548d1b43084bbc94fc3e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:13.129712 463 master_service.cc:946] ReplaceTablet: received request to replace tablet 83b5b36a26e74668998a19fa0ddf7188 from {username='slave'} at 127.0.0.1:33540
I20250114 02:54:13.135893 463 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_3497e36f166142c3a1ef161324c8f45f" } modify_external_catalogs: true
I20250114 02:54:13.136463 463 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_3497e36f166142c3a1ef161324c8f45f" } modify_external_catalogs: true
I20250114 02:54:13.140079 1206 tablet_bootstrap.cc:492] T 276942e5e54246c79f0d4f08297f58ea P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:13.140532 1206 ts_tablet_manager.cc:1397] T 276942e5e54246c79f0d4f08297f58ea P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.381s user 0.011s sys 0.003s
I20250114 02:54:13.142654 1206 raft_consensus.cc:357] T 276942e5e54246c79f0d4f08297f58ea P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:13.146342 1206 raft_consensus.cc:383] T 276942e5e54246c79f0d4f08297f58ea P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:13.146634 1206 raft_consensus.cc:738] T 276942e5e54246c79f0d4f08297f58ea P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:13.147329 1206 consensus_queue.cc:260] T 276942e5e54246c79f0d4f08297f58ea P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:13.150128 1105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b4b6d4ac364a14a34733b4dd715bb3" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:13.150683 1105 raft_consensus.cc:3054] T 61b4b6d4ac364a14a34733b4dd715bb3 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:13.153584 1205 tablet_bootstrap.cc:492] T 4bb9b429af92446bbe13894d5dfe4c47 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:13.153944 1205 ts_tablet_manager.cc:1397] T 4bb9b429af92446bbe13894d5dfe4c47 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.247s user 0.010s sys 0.000s
I20250114 02:54:13.153906 467 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_ed523e66ce6945a7a0fcdb96e498131d" } modify_external_catalogs: true
I20250114 02:54:13.154438 467 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_ed523e66ce6945a7a0fcdb96e498131d" } modify_external_catalogs: true
I20250114 02:54:13.159404 1206 ts_tablet_manager.cc:1428] T 276942e5e54246c79f0d4f08297f58ea P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.019s user 0.004s sys 0.000s
I20250114 02:54:13.161036 1105 raft_consensus.cc:2463] T 61b4b6d4ac364a14a34733b4dd715bb3 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 1.
I20250114 02:54:13.162214 753 leader_election.cc:304] T 61b4b6d4ac364a14a34733b4dd715bb3 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:13.163041 1415 raft_consensus.cc:2798] T 61b4b6d4ac364a14a34733b4dd715bb3 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:13.163450 1415 raft_consensus.cc:695] T 61b4b6d4ac364a14a34733b4dd715bb3 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 LEADER]: Becoming Leader. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Running, Role: LEADER
I20250114 02:54:13.164554 1439 tablet.cc:1721] T d434cbc9daf549a6bdc04c8c76ead8a6 P 525198593c8e47afaee54e74267b87e3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250114 02:54:13.164146 1415 consensus_queue.cc:237] T 61b4b6d4ac364a14a34733b4dd715bb3 P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:13.174459 1424 consensus_queue.cc:1035] T 857119ca5f9d4ad98f899df22f32244b P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:54:13.179530 1207 tablet_bootstrap.cc:700] T 83b5b36a26e74668998a19fa0ddf7188 P 90d6c2c58bb54e0eafb365625ac33871: Time spent opening tablet: real 0.267s user 0.002s sys 0.000s
I20250114 02:54:13.184701 1442 tablet.cc:1721] T a3886c5f976445cebc95ef1839a74aa9 P 525198593c8e47afaee54e74267b87e3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250114 02:54:13.186379 1235 raft_consensus.cc:491] T 8b4bc535c42e43889401f14b6d59659e P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:13.186802 1235 raft_consensus.cc:513] T 8b4bc535c42e43889401f14b6d59659e P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:13.187400 1435 tablet.cc:1721] T a3886c5f976445cebc95ef1839a74aa9 P 90d6c2c58bb54e0eafb365625ac33871: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250114 02:54:13.189313 1235 leader_election.cc:290] T 8b4bc535c42e43889401f14b6d59659e P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:13.191821 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b4bc535c42e43889401f14b6d59659e" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:13.191874 960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b4b6d4ac364a14a34733b4dd715bb3" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
I20250114 02:54:13.192471 1389 leader_election.cc:290] T 61b4b6d4ac364a14a34733b4dd715bb3 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Requested vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:13.192569 960 raft_consensus.cc:3054] T 61b4b6d4ac364a14a34733b4dd715bb3 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
W20250114 02:54:13.193150 898 leader_election.cc:343] T 8b4bc535c42e43889401f14b6d59659e P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:13.180152 1207 tablet_bootstrap.cc:654] T 83b5b36a26e74668998a19fa0ddf7188 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:13.194150 1105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b4bc535c42e43889401f14b6d59659e" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:13.194676 1105 raft_consensus.cc:2463] T 8b4bc535c42e43889401f14b6d59659e P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:13.195648 898 leader_election.cc:304] T 8b4bc535c42e43889401f14b6d59659e P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters: 819f9e3412e54d44abb11d8b3ca877c2
I20250114 02:54:13.196630 1235 raft_consensus.cc:2798] T 8b4bc535c42e43889401f14b6d59659e P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:13.196931 1235 raft_consensus.cc:491] T 8b4bc535c42e43889401f14b6d59659e P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:13.197240 1206 tablet_bootstrap.cc:492] T 8b4bc535c42e43889401f14b6d59659e P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:13.197362 1235 raft_consensus.cc:3054] T 8b4bc535c42e43889401f14b6d59659e P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:13.201557 1433 tablet.cc:1721] T d434cbc9daf549a6bdc04c8c76ead8a6 P 819f9e3412e54d44abb11d8b3ca877c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250114 02:54:13.155929 1205 raft_consensus.cc:357] T 4bb9b429af92446bbe13894d5dfe4c47 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:13.205803 1436 tablet.cc:1721] T d434cbc9daf549a6bdc04c8c76ead8a6 P 90d6c2c58bb54e0eafb365625ac33871: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250114 02:54:13.206732 1205 raft_consensus.cc:383] T 4bb9b429af92446bbe13894d5dfe4c47 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:13.211668 1223 ts_tablet_manager.cc:1918] T 0752ab4b7031402db4e6be8b61cf364f P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:13.212000 1223 log.cc:1198] T 0752ab4b7031402db4e6be8b61cf364f P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/0752ab4b7031402db4e6be8b61cf364f
I20250114 02:54:13.212596 1223 ts_tablet_manager.cc:1939] T 0752ab4b7031402db4e6be8b61cf364f P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
I20250114 02:54:13.214721 462 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_4da7342e02e441008994800e5215bcae" } new_table_name: "default.Table_8344ae9b78e64a038ba7e64f1f807946" modify_external_catalogs: true
I20250114 02:54:13.221803 1205 raft_consensus.cc:738] T 4bb9b429af92446bbe13894d5dfe4c47 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:13.222467 1205 consensus_queue.cc:260] T 4bb9b429af92446bbe13894d5dfe4c47 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:13.224475 1205 ts_tablet_manager.cc:1428] T 4bb9b429af92446bbe13894d5dfe4c47 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.070s user 0.007s sys 0.000s
I20250114 02:54:13.226007 1205 tablet_bootstrap.cc:492] T 83b5b36a26e74668998a19fa0ddf7188 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:13.227586 1207 tablet_bootstrap.cc:492] T 83b5b36a26e74668998a19fa0ddf7188 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:13.236898 1394 ts_tablet_manager.cc:1918] T 0752ab4b7031402db4e6be8b61cf364f P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:13.237447 1394 log.cc:1198] T 0752ab4b7031402db4e6be8b61cf364f P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/0752ab4b7031402db4e6be8b61cf364f
I20250114 02:54:13.238061 1394 ts_tablet_manager.cc:1939] T 0752ab4b7031402db4e6be8b61cf364f P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:13.240279 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet 0752ab4b7031402db4e6be8b61cf364f (table default.Table_e691c983a1bc4ce3a3d4ac683c048218 [id=0543f6180069494c8055f8931b9ac3c6]) successfully deleted
I20250114 02:54:13.235702 1206 tablet_bootstrap.cc:654] T 8b4bc535c42e43889401f14b6d59659e P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:13.248163 1207 ts_tablet_manager.cc:1397] T 83b5b36a26e74668998a19fa0ddf7188 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.371s user 0.010s sys 0.000s
I20250114 02:54:13.254161 1207 raft_consensus.cc:357] T 83b5b36a26e74668998a19fa0ddf7188 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:13.257740 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet 0752ab4b7031402db4e6be8b61cf364f (table default.Table_e691c983a1bc4ce3a3d4ac683c048218 [id=0543f6180069494c8055f8931b9ac3c6]) successfully deleted
I20250114 02:54:13.254827 1207 raft_consensus.cc:383] T 83b5b36a26e74668998a19fa0ddf7188 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:13.262521 1207 raft_consensus.cc:738] T 83b5b36a26e74668998a19fa0ddf7188 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:13.258529 1223 tablet_replica.cc:331] T 857119ca5f9d4ad98f899df22f32244b P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:13.263494 1223 raft_consensus.cc:2238] T 857119ca5f9d4ad98f899df22f32244b P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:13.263895 1223 raft_consensus.cc:2267] T 857119ca5f9d4ad98f899df22f32244b P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:13.265010 1207 consensus_queue.cc:260] T 83b5b36a26e74668998a19fa0ddf7188 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:13.270151 1207 ts_tablet_manager.cc:1428] T 83b5b36a26e74668998a19fa0ddf7188 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.022s user 0.005s sys 0.000s
I20250114 02:54:13.270480 464 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet c20aca761313498193e1d79c10705f8f
I20250114 02:54:13.270960 1207 tablet_bootstrap.cc:492] T b63dd90f60a04465b948171e74888e79 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:13.271610 464 catalog_manager.cc:6596] ReplaceTablet: tablet c20aca761313498193e1d79c10705f8f deleted and replaced by tablet bc46815b4803418c95ea9c039a4fae88
I20250114 02:54:13.273615 1223 ts_tablet_manager.cc:1905] T 857119ca5f9d4ad98f899df22f32244b P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:13.274787 465 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 61b4b6d4ac364a14a34733b4dd715bb3 (table default.Table_0931c459ae794d2091fcc0f51d031548 [id=e279c31bd63b40b9ade03a3b928bf153]). Expected version 2 got 0
I20250114 02:54:13.275357 465 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 8b4bc535c42e43889401f14b6d59659e (table default.Table_d90b2995b13b4a8fb7e50cc4b79cbf84 [id=f8e40e0aa44b4157ac3438d11685b2e7]). Expected version 1 got 0
I20250114 02:54:13.277091 1376 consensus_queue.cc:1035] T 92b8d48880434ab99eec7f1e452931fa P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:13.284752 794 tablet_service.cc:1514] Processing DeleteTablet for tablet c20aca761313498193e1d79c10705f8f with delete_type TABLET_DATA_DELETED (replaced by tablet bc46815b4803418c95ea9c039a4fae88) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:13.295089 1207 tablet_bootstrap.cc:654] T b63dd90f60a04465b948171e74888e79 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:13.275882 465 catalog_manager.cc:5526] T b3b1ea0a825c4d1f8e3edf24cc74d8f7 P 525198593c8e47afaee54e74267b87e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 525198593c8e47afaee54e74267b87e3 (127.0.107.66). New cstate: current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } }
I20250114 02:54:13.299182 465 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet d434cbc9daf549a6bdc04c8c76ead8a6 (table default.Table_003a5067105b43fdaba7a58fbd24bc3a [id=5e6aa799d6444bad968f29327ebace9b]). Expected version 1 got 0
I20250114 02:54:13.309499 1223 ts_tablet_manager.cc:1918] T 857119ca5f9d4ad98f899df22f32244b P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:13.311627 1223 log.cc:1198] T 857119ca5f9d4ad98f899df22f32244b P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/857119ca5f9d4ad98f899df22f32244b
I20250114 02:54:13.311683 1394 tablet_replica.cc:331] T 857119ca5f9d4ad98f899df22f32244b P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:13.312384 1223 ts_tablet_manager.cc:1939] T 857119ca5f9d4ad98f899df22f32244b P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
I20250114 02:54:13.312548 1394 raft_consensus.cc:2238] T 857119ca5f9d4ad98f899df22f32244b P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 LEADER]: Raft consensus shutting down.
I20250114 02:54:13.314934 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet 857119ca5f9d4ad98f899df22f32244b (table default.Table_55dad15f1b4f4e22854ba3070b0eea8e [id=5f5b8d19482b4137add134b196bfd20d]) successfully deleted
I20250114 02:54:13.316892 1394 raft_consensus.cc:2267] T 857119ca5f9d4ad98f899df22f32244b P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:13.317117 465 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet a3886c5f976445cebc95ef1839a74aa9 (table default.Table_35a78676616e4b20906b262c9d0c0d9e [id=e984149e9da444f1b2802e3f35a60fd6]). Expected version 2 got 0
I20250114 02:54:13.317761 465 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet b63dd90f60a04465b948171e74888e79 (table default.Table_1097f09fb10e441fb639f8d7aa12d487 [id=d8c9691818d042d7b49475fa009c4ca1]). Expected version 3 got 0
I20250114 02:54:13.319239 1394 ts_tablet_manager.cc:1905] T 857119ca5f9d4ad98f899df22f32244b P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:13.325686 1206 tablet_bootstrap.cc:492] T 8b4bc535c42e43889401f14b6d59659e P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:13.325778 1081 tablet_service.cc:1514] Processing DeleteTablet for tablet c20aca761313498193e1d79c10705f8f with delete_type TABLET_DATA_DELETED (replaced by tablet bc46815b4803418c95ea9c039a4fae88) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:13.326074 1206 ts_tablet_manager.cc:1397] T 8b4bc535c42e43889401f14b6d59659e P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.129s user 0.011s sys 0.000s
I20250114 02:54:13.203821 1235 raft_consensus.cc:513] T 8b4bc535c42e43889401f14b6d59659e P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:13.332407 1235 leader_election.cc:290] T 8b4bc535c42e43889401f14b6d59659e P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Requested vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:13.333624 1386 consensus_queue.cc:1035] T 92b8d48880434ab99eec7f1e452931fa P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250114 02:54:13.335322 458 catalog_manager.cc:4852] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet c20aca761313498193e1d79c10705f8f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c20aca761313498193e1d79c10705f8f already in progress: creating tablet
I20250114 02:54:13.335403 817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b4bc535c42e43889401f14b6d59659e" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
W20250114 02:54:13.336452 898 leader_election.cc:343] T 8b4bc535c42e43889401f14b6d59659e P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:13.337083 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b4bc535c42e43889401f14b6d59659e" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:13.337577 1106 raft_consensus.cc:3054] T 8b4bc535c42e43889401f14b6d59659e P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:13.337535 1206 raft_consensus.cc:357] T 8b4bc535c42e43889401f14b6d59659e P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:13.338141 1206 raft_consensus.cc:383] T 8b4bc535c42e43889401f14b6d59659e P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:13.338383 1206 raft_consensus.cc:738] T 8b4bc535c42e43889401f14b6d59659e P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:13.221272 960 raft_consensus.cc:2463] T 61b4b6d4ac364a14a34733b4dd715bb3 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 1.
I20250114 02:54:13.261436 1393 ts_tablet_manager.cc:1918] T 0752ab4b7031402db4e6be8b61cf364f P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:13.340708 1393 log.cc:1198] T 0752ab4b7031402db4e6be8b61cf364f P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/0752ab4b7031402db4e6be8b61cf364f
I20250114 02:54:13.341480 1393 ts_tablet_manager.cc:1939] T 0752ab4b7031402db4e6be8b61cf364f P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:13.345113 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet 0752ab4b7031402db4e6be8b61cf364f (table default.Table_e691c983a1bc4ce3a3d4ac683c048218 [id=0543f6180069494c8055f8931b9ac3c6]) successfully deleted
I20250114 02:54:13.279713 936 tablet_service.cc:1514] Processing DeleteTablet for tablet c20aca761313498193e1d79c10705f8f with delete_type TABLET_DATA_DELETED (replaced by tablet bc46815b4803418c95ea9c039a4fae88) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:13.346172 1206 consensus_queue.cc:260] T 8b4bc535c42e43889401f14b6d59659e P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:13.356348 1393 tablet_replica.cc:331] T 857119ca5f9d4ad98f899df22f32244b P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:13.357367 1393 raft_consensus.cc:2238] T 857119ca5f9d4ad98f899df22f32244b P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:13.357801 1393 raft_consensus.cc:2267] T 857119ca5f9d4ad98f899df22f32244b P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:13.359540 1206 ts_tablet_manager.cc:1428] T 8b4bc535c42e43889401f14b6d59659e P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.033s user 0.006s sys 0.001s
I20250114 02:54:13.364171 1106 raft_consensus.cc:2463] T 8b4bc535c42e43889401f14b6d59659e P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:13.365132 1207 tablet_bootstrap.cc:492] T b63dd90f60a04465b948171e74888e79 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:13.365541 1207 ts_tablet_manager.cc:1397] T b63dd90f60a04465b948171e74888e79 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.095s user 0.012s sys 0.026s
I20250114 02:54:13.365715 898 leader_election.cc:304] T 8b4bc535c42e43889401f14b6d59659e P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters: 819f9e3412e54d44abb11d8b3ca877c2
I20250114 02:54:13.366788 1235 raft_consensus.cc:2798] T 8b4bc535c42e43889401f14b6d59659e P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:13.367224 1235 raft_consensus.cc:695] T 8b4bc535c42e43889401f14b6d59659e P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Becoming Leader. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Running, Role: LEADER
I20250114 02:54:13.367956 1235 consensus_queue.cc:237] T 8b4bc535c42e43889401f14b6d59659e P 525198593c8e47afaee54e74267b87e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:13.360271 1206 tablet_bootstrap.cc:492] T b3b1ea0a825c4d1f8e3edf24cc74d8f7 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:13.374014 1393 ts_tablet_manager.cc:1905] T 857119ca5f9d4ad98f899df22f32244b P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:13.383503 1408 raft_consensus.cc:491] T b63dd90f60a04465b948171e74888e79 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:13.384001 1408 raft_consensus.cc:513] T b63dd90f60a04465b948171e74888e79 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:13.385835 1408 leader_election.cc:290] T b63dd90f60a04465b948171e74888e79 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:13.386909 1205 tablet_bootstrap.cc:654] T 83b5b36a26e74668998a19fa0ddf7188 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:13.387970 1207 raft_consensus.cc:357] T b63dd90f60a04465b948171e74888e79 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:13.388790 1207 raft_consensus.cc:383] T b63dd90f60a04465b948171e74888e79 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:13.388557 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b63dd90f60a04465b948171e74888e79" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:13.395637 1207 raft_consensus.cc:738] T b63dd90f60a04465b948171e74888e79 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:13.391978 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b63dd90f60a04465b948171e74888e79" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
W20250114 02:54:13.390220 898 leader_election.cc:343] T b63dd90f60a04465b948171e74888e79 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:13.398051 1206 tablet_bootstrap.cc:654] T b3b1ea0a825c4d1f8e3edf24cc74d8f7 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:13.396409 1207 consensus_queue.cc:260] T b63dd90f60a04465b948171e74888e79 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:13.401929 1106 raft_consensus.cc:2463] T b63dd90f60a04465b948171e74888e79 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:13.404908 898 leader_election.cc:304] T b63dd90f60a04465b948171e74888e79 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters: 819f9e3412e54d44abb11d8b3ca877c2
I20250114 02:54:13.406795 470 master_service.cc:946] ReplaceTablet: received request to replace tablet 61b4b6d4ac364a14a34733b4dd715bb3 from {username='slave'} at 127.0.0.1:33540
I20250114 02:54:13.407662 1207 ts_tablet_manager.cc:1428] T b63dd90f60a04465b948171e74888e79 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.042s user 0.005s sys 0.000s
I20250114 02:54:13.408529 1207 tablet_bootstrap.cc:492] T 268a12df225e4ba59d4a84c2869d4661 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:13.412783 1235 raft_consensus.cc:2798] T b63dd90f60a04465b948171e74888e79 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:13.413167 1235 raft_consensus.cc:491] T b63dd90f60a04465b948171e74888e79 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:13.413511 1235 raft_consensus.cc:3054] T b63dd90f60a04465b948171e74888e79 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:13.418169 464 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_150e45eea89e48369a240547174217ef" } new_table_name: "default.Table_4fd891b63924429085d78f78cb177f0e" modify_external_catalogs: true
I20250114 02:54:13.420641 1393 ts_tablet_manager.cc:1918] T 857119ca5f9d4ad98f899df22f32244b P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:13.420984 1393 log.cc:1198] T 857119ca5f9d4ad98f899df22f32244b P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/857119ca5f9d4ad98f899df22f32244b
I20250114 02:54:13.421247 467 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet 9589afd0094545c1801b7a0b5eb8698f
I20250114 02:54:13.421753 1393 ts_tablet_manager.cc:1939] T 857119ca5f9d4ad98f899df22f32244b P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:13.423060 794 tablet_service.cc:1514] Processing DeleteTablet for tablet 9589afd0094545c1801b7a0b5eb8698f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:13 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:13.423532 936 tablet_service.cc:1514] Processing DeleteTablet for tablet 9589afd0094545c1801b7a0b5eb8698f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:13 UTC) from {username='slave'} at 127.0.0.1:55068
W20250114 02:54:13.423738 462 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet aad96f2507974023aaa988e6950feb7a has no consensus state
I20250114 02:54:13.423100 1086 tablet_service.cc:1514] Processing DeleteTablet for tablet 9589afd0094545c1801b7a0b5eb8698f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:13 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:13.430505 471 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_42c4106f7e6644d79bb958e314e449e5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:13.431814 471 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_42c4106f7e6644d79bb958e314e449e5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:13.432565 1207 tablet_bootstrap.cc:654] T 268a12df225e4ba59d4a84c2869d4661 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
W20250114 02:54:13.435580 456 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 179248cb5acf4740b268e0ec8b2852f9 (table default.Table_bd405598fe054b718a8d6fb5dd875f0e [id=5745501ecd7b4cecaca3d35502a6cc40], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 179248cb5acf4740b268e0ec8b2852f9 consensus state has no leader
I20250114 02:54:13.438344 1394 ts_tablet_manager.cc:1918] T 857119ca5f9d4ad98f899df22f32244b P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:13.438704 1394 log.cc:1198] T 857119ca5f9d4ad98f899df22f32244b P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/857119ca5f9d4ad98f899df22f32244b
I20250114 02:54:13.439380 1394 ts_tablet_manager.cc:1939] T 857119ca5f9d4ad98f899df22f32244b P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:13.444044 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet 857119ca5f9d4ad98f899df22f32244b (table default.Table_55dad15f1b4f4e22854ba3070b0eea8e [id=5f5b8d19482b4137add134b196bfd20d]) successfully deleted
W20250114 02:54:13.444674 456 catalog_manager.cc:4852] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet c20aca761313498193e1d79c10705f8f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c20aca761313498193e1d79c10705f8f already in progress: creating tablet
W20250114 02:54:13.446574 458 catalog_manager.cc:4852] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 9589afd0094545c1801b7a0b5eb8698f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9589afd0094545c1801b7a0b5eb8698f already in progress: creating tablet
W20250114 02:54:13.459614 456 catalog_manager.cc:4852] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet 9589afd0094545c1801b7a0b5eb8698f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9589afd0094545c1801b7a0b5eb8698f already in progress: creating tablet
I20250114 02:54:13.467893 463 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet 92b8d48880434ab99eec7f1e452931fa
I20250114 02:54:13.474440 794 tablet_service.cc:1514] Processing DeleteTablet for tablet 92b8d48880434ab99eec7f1e452931fa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:13 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:13.475546 1086 tablet_service.cc:1514] Processing DeleteTablet for tablet 92b8d48880434ab99eec7f1e452931fa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:13 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:13.476369 1223 tablet_replica.cc:331] T 92b8d48880434ab99eec7f1e452931fa P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:13.476514 1394 tablet_replica.cc:331] T 92b8d48880434ab99eec7f1e452931fa P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:13.477061 1223 raft_consensus.cc:2238] T 92b8d48880434ab99eec7f1e452931fa P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Raft consensus shutting down.
I20250114 02:54:13.477277 1394 raft_consensus.cc:2238] T 92b8d48880434ab99eec7f1e452931fa P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:13.477717 1394 raft_consensus.cc:2267] T 92b8d48880434ab99eec7f1e452931fa P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:13.477977 1223 raft_consensus.cc:2267] T 92b8d48880434ab99eec7f1e452931fa P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:13.479938 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet 857119ca5f9d4ad98f899df22f32244b (table default.Table_55dad15f1b4f4e22854ba3070b0eea8e [id=5f5b8d19482b4137add134b196bfd20d]) successfully deleted
W20250114 02:54:13.480676 458 catalog_manager.cc:4852] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet c20aca761313498193e1d79c10705f8f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c20aca761313498193e1d79c10705f8f already in progress: creating tablet
W20250114 02:54:13.481313 458 catalog_manager.cc:4852] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet 9589afd0094545c1801b7a0b5eb8698f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9589afd0094545c1801b7a0b5eb8698f already in progress: creating tablet
I20250114 02:54:13.482975 1086 tablet_service.cc:1514] Processing DeleteTablet for tablet 0752ab4b7031402db4e6be8b61cf364f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:12 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:13.485280 1223 ts_tablet_manager.cc:1905] T 92b8d48880434ab99eec7f1e452931fa P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:13.485600 1394 ts_tablet_manager.cc:1905] T 92b8d48880434ab99eec7f1e452931fa P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:13.490744 462 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_5d3b333f1d99463b8ad3b4318c8f29df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:13.493197 940 tablet_service.cc:1514] Processing DeleteTablet for tablet 92b8d48880434ab99eec7f1e452931fa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:13 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:13.494196 1393 tablet_replica.cc:331] T 92b8d48880434ab99eec7f1e452931fa P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:13.504103 469 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet a3886c5f976445cebc95ef1839a74aa9 (table default.Table_35a78676616e4b20906b262c9d0c0d9e [id=e984149e9da444f1b2802e3f35a60fd6]). Expected version 2 got 0
I20250114 02:54:13.504827 469 catalog_manager.cc:5526] T b4740f6b3a6741368ba68907ad065279 P 819f9e3412e54d44abb11d8b3ca877c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65). New cstate: current_term: 1 leader_uuid: "819f9e3412e54d44abb11d8b3ca877c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:13.506165 469 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 4bb9b429af92446bbe13894d5dfe4c47 (table default.Table_acb5bf9354be440bbf6034cf1c535dab [id=22b7edcf4b7e4c31b3c23dc44ccf028e]). Expected version 1 got 0
I20250114 02:54:13.506646 469 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet d434cbc9daf549a6bdc04c8c76ead8a6 (table default.Table_003a5067105b43fdaba7a58fbd24bc3a [id=5e6aa799d6444bad968f29327ebace9b]). Expected version 1 got 0
W20250114 02:54:13.508620 464 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f464eabfbc0945fa8a5583f00c1dab40 has no consensus state
I20250114 02:54:13.508713 1223 ts_tablet_manager.cc:1918] T 92b8d48880434ab99eec7f1e452931fa P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250114 02:54:13.511276 462 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5d3b333f1d99463b8ad3b4318c8f29df in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:13.513332 466 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_ae6b6fdfc1834cfb8089c860aedcc964"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:13.515906 940 tablet_service.cc:1514] Processing DeleteTablet for tablet 0752ab4b7031402db4e6be8b61cf364f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:12 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:13.520366 1393 raft_consensus.cc:2238] T 92b8d48880434ab99eec7f1e452931fa P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:13.521093 1393 raft_consensus.cc:2267] T 92b8d48880434ab99eec7f1e452931fa P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:13.521606 1223 log.cc:1198] T 92b8d48880434ab99eec7f1e452931fa P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/92b8d48880434ab99eec7f1e452931fa
I20250114 02:54:13.522356 1223 ts_tablet_manager.cc:1939] T 92b8d48880434ab99eec7f1e452931fa P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
I20250114 02:54:13.524590 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet 92b8d48880434ab99eec7f1e452931fa (table default.Table_3497e36f166142c3a1ef161324c8f45f [id=a3a41d4e0b004a4aaefc1c1009a8f244]) successfully deleted
I20250114 02:54:13.528576 1393 ts_tablet_manager.cc:1905] T 92b8d48880434ab99eec7f1e452931fa P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:13.530624 470 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet 61b4b6d4ac364a14a34733b4dd715bb3
I20250114 02:54:13.531780 470 catalog_manager.cc:6596] ReplaceTablet: tablet 61b4b6d4ac364a14a34733b4dd715bb3 deleted and replaced by tablet 6448eea21ab2422c97da2494658ded3f
I20250114 02:54:13.532796 1086 tablet_service.cc:1514] Processing DeleteTablet for tablet 61b4b6d4ac364a14a34733b4dd715bb3 with delete_type TABLET_DATA_DELETED (replaced by tablet 6448eea21ab2422c97da2494658ded3f) from {username='slave'} at 127.0.0.1:53876
W20250114 02:54:13.514730 466 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ae6b6fdfc1834cfb8089c860aedcc964 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:13.534113 794 tablet_service.cc:1514] Processing DeleteTablet for tablet 61b4b6d4ac364a14a34733b4dd715bb3 with delete_type TABLET_DATA_DELETED (replaced by tablet 6448eea21ab2422c97da2494658ded3f) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:13.534984 940 tablet_service.cc:1514] Processing DeleteTablet for tablet 61b4b6d4ac364a14a34733b4dd715bb3 with delete_type TABLET_DATA_DELETED (replaced by tablet 6448eea21ab2422c97da2494658ded3f) from {username='slave'} at 127.0.0.1:55068
W20250114 02:54:13.535291 458 catalog_manager.cc:4839] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 0752ab4b7031402db4e6be8b61cf364f because the tablet was not found. No further retry: Not found: Tablet not found: 0752ab4b7031402db4e6be8b61cf364f
I20250114 02:54:13.536665 1223 tablet_replica.cc:331] T 61b4b6d4ac364a14a34733b4dd715bb3 P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:13.538364 1223 raft_consensus.cc:2238] T 61b4b6d4ac364a14a34733b4dd715bb3 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:13.538866 1223 raft_consensus.cc:2267] T 61b4b6d4ac364a14a34733b4dd715bb3 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:13.540587 1223 ts_tablet_manager.cc:1905] T 61b4b6d4ac364a14a34733b4dd715bb3 P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:13.547132 470 master_service.cc:946] ReplaceTablet: received request to replace tablet 7ecf782bb3d54528aa996abca1ea3a15 from {username='slave'} at 127.0.0.1:33540
I20250114 02:54:13.549296 1394 ts_tablet_manager.cc:1918] T 92b8d48880434ab99eec7f1e452931fa P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:13.549638 1394 log.cc:1198] T 92b8d48880434ab99eec7f1e452931fa P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/92b8d48880434ab99eec7f1e452931fa
I20250114 02:54:13.550256 1394 ts_tablet_manager.cc:1939] T 92b8d48880434ab99eec7f1e452931fa P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:13.551291 1394 tablet_replica.cc:331] T 61b4b6d4ac364a14a34733b4dd715bb3 P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:13.551431 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet 92b8d48880434ab99eec7f1e452931fa (table default.Table_3497e36f166142c3a1ef161324c8f45f [id=a3a41d4e0b004a4aaefc1c1009a8f244]) successfully deleted
I20250114 02:54:13.561532 1386 raft_consensus.cc:491] T 83b5b36a26e74668998a19fa0ddf7188 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:13.562484 1386 raft_consensus.cc:513] T 83b5b36a26e74668998a19fa0ddf7188 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:13.565187 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83b5b36a26e74668998a19fa0ddf7188" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:13.565822 1386 leader_election.cc:290] T 83b5b36a26e74668998a19fa0ddf7188 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:13.566792 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83b5b36a26e74668998a19fa0ddf7188" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
W20250114 02:54:13.567107 1042 leader_election.cc:343] T 83b5b36a26e74668998a19fa0ddf7188 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:13.560917 1235 raft_consensus.cc:513] T b63dd90f60a04465b948171e74888e79 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
W20250114 02:54:13.568259 1040 leader_election.cc:343] T 83b5b36a26e74668998a19fa0ddf7188 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:13.568608 1040 leader_election.cc:304] T 83b5b36a26e74668998a19fa0ddf7188 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90d6c2c58bb54e0eafb365625ac33871; no voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2
I20250114 02:54:13.569975 1386 raft_consensus.cc:2743] T 83b5b36a26e74668998a19fa0ddf7188 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:13.570230 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b63dd90f60a04465b948171e74888e79" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
W20250114 02:54:13.571417 898 leader_election.cc:343] T b63dd90f60a04465b948171e74888e79 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:13.572928 470 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet 7ecf782bb3d54528aa996abca1ea3a15
I20250114 02:54:13.574155 470 catalog_manager.cc:6596] ReplaceTablet: tablet 7ecf782bb3d54528aa996abca1ea3a15 deleted and replaced by tablet 66bfa92db9874750b3c340e31a06c412
I20250114 02:54:13.574908 794 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ecf782bb3d54528aa996abca1ea3a15 with delete_type TABLET_DATA_DELETED (replaced by tablet 66bfa92db9874750b3c340e31a06c412) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:13.552022 1394 raft_consensus.cc:2238] T 61b4b6d4ac364a14a34733b4dd715bb3 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 LEADER]: Raft consensus shutting down.
I20250114 02:54:13.575747 940 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ecf782bb3d54528aa996abca1ea3a15 with delete_type TABLET_DATA_DELETED (replaced by tablet 66bfa92db9874750b3c340e31a06c412) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:13.575978 1086 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ecf782bb3d54528aa996abca1ea3a15 with delete_type TABLET_DATA_DELETED (replaced by tablet 66bfa92db9874750b3c340e31a06c412) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:13.580296 1394 pending_rounds.cc:62] T 61b4b6d4ac364a14a34733b4dd715bb3 P 819f9e3412e54d44abb11d8b3ca877c2: Trying to abort 1 pending ops.
I20250114 02:54:13.580595 1394 pending_rounds.cc:69] T 61b4b6d4ac364a14a34733b4dd715bb3 P 819f9e3412e54d44abb11d8b3ca877c2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114028044970098688 op_type: NO_OP noop_request { }
I20250114 02:54:13.581010 1394 raft_consensus.cc:2883] T 61b4b6d4ac364a14a34733b4dd715bb3 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250114 02:54:13.581347 1394 raft_consensus.cc:2267] T 61b4b6d4ac364a14a34733b4dd715bb3 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:13.588968 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b63dd90f60a04465b948171e74888e79" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:13.589576 1106 raft_consensus.cc:3054] T b63dd90f60a04465b948171e74888e79 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:13.597312 1106 raft_consensus.cc:2463] T b63dd90f60a04465b948171e74888e79 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:13.598026 1394 ts_tablet_manager.cc:1905] T 61b4b6d4ac364a14a34733b4dd715bb3 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:13.598598 898 leader_election.cc:304] T b63dd90f60a04465b948171e74888e79 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters: 819f9e3412e54d44abb11d8b3ca877c2
I20250114 02:54:13.599334 1408 raft_consensus.cc:2798] T b63dd90f60a04465b948171e74888e79 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:13.599615 1223 ts_tablet_manager.cc:1918] T 61b4b6d4ac364a14a34733b4dd715bb3 P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:13.603252 1393 ts_tablet_manager.cc:1918] T 92b8d48880434ab99eec7f1e452931fa P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:13.603539 1223 log.cc:1198] T 61b4b6d4ac364a14a34733b4dd715bb3 P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/61b4b6d4ac364a14a34733b4dd715bb3
I20250114 02:54:13.603610 1393 log.cc:1198] T 92b8d48880434ab99eec7f1e452931fa P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/92b8d48880434ab99eec7f1e452931fa
I20250114 02:54:13.604156 1223 ts_tablet_manager.cc:1939] T 61b4b6d4ac364a14a34733b4dd715bb3 P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
I20250114 02:54:13.605952 1393 ts_tablet_manager.cc:1939] T 92b8d48880434ab99eec7f1e452931fa P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:13.607483 1235 leader_election.cc:290] T b63dd90f60a04465b948171e74888e79 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Requested vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:13.607640 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet 61b4b6d4ac364a14a34733b4dd715bb3 (table default.Table_0931c459ae794d2091fcc0f51d031548 [id=e279c31bd63b40b9ade03a3b928bf153]) successfully deleted
I20250114 02:54:13.617518 1206 tablet_bootstrap.cc:492] T b3b1ea0a825c4d1f8e3edf24cc74d8f7 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:13.617982 1206 ts_tablet_manager.cc:1397] T b3b1ea0a825c4d1f8e3edf24cc74d8f7 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.258s user 0.013s sys 0.005s
I20250114 02:54:13.618970 1393 tablet_replica.cc:331] T 61b4b6d4ac364a14a34733b4dd715bb3 P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:13.620445 1205 tablet_bootstrap.cc:492] T 83b5b36a26e74668998a19fa0ddf7188 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:13.621131 1205 ts_tablet_manager.cc:1397] T 83b5b36a26e74668998a19fa0ddf7188 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.395s user 0.036s sys 0.029s
W20250114 02:54:13.623684 458 catalog_manager.cc:4852] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 7ecf782bb3d54528aa996abca1ea3a15 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7ecf782bb3d54528aa996abca1ea3a15 already in progress: creating tablet
I20250114 02:54:13.623829 1205 raft_consensus.cc:357] T 83b5b36a26e74668998a19fa0ddf7188 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:13.624468 1205 raft_consensus.cc:383] T 83b5b36a26e74668998a19fa0ddf7188 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:13.624711 1205 raft_consensus.cc:738] T 83b5b36a26e74668998a19fa0ddf7188 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:13.625247 1205 consensus_queue.cc:260] T 83b5b36a26e74668998a19fa0ddf7188 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:13.626921 1207 tablet_bootstrap.cc:492] T 268a12df225e4ba59d4a84c2869d4661 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:13.627329 1207 ts_tablet_manager.cc:1397] T 268a12df225e4ba59d4a84c2869d4661 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.219s user 0.026s sys 0.025s
I20250114 02:54:13.627476 466 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_02b3fc3bfe484e3a9c5d5e9114deb368" } modify_external_catalogs: true
I20250114 02:54:13.627245 469 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_ae220acb7e0345bdbb80b11398e4d153"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:13.628162 462 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_47464943699c413b9c925aa383bfe64a" } new_table_name: "default.Table_b6330aba5846413c915c50dc4c6e5f11" modify_external_catalogs: true
I20250114 02:54:13.628497 1393 raft_consensus.cc:2238] T 61b4b6d4ac364a14a34733b4dd715bb3 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:13.628914 1393 raft_consensus.cc:2267] T 61b4b6d4ac364a14a34733b4dd715bb3 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:13.628794 466 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_02b3fc3bfe484e3a9c5d5e9114deb368" } modify_external_catalogs: true
I20250114 02:54:13.629885 1207 raft_consensus.cc:357] T 268a12df225e4ba59d4a84c2869d4661 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:13.630508 1207 raft_consensus.cc:383] T 268a12df225e4ba59d4a84c2869d4661 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:13.624362 1408 raft_consensus.cc:695] T b63dd90f60a04465b948171e74888e79 P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Becoming Leader. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Running, Role: LEADER
I20250114 02:54:13.630746 1207 raft_consensus.cc:738] T 268a12df225e4ba59d4a84c2869d4661 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:13.631283 1207 consensus_queue.cc:260] T 268a12df225e4ba59d4a84c2869d4661 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:13.631791 1408 consensus_queue.cc:237] T b63dd90f60a04465b948171e74888e79 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
W20250114 02:54:13.636407 469 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ae220acb7e0345bdbb80b11398e4d153 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:13.621099 1206 raft_consensus.cc:357] T b3b1ea0a825c4d1f8e3edf24cc74d8f7 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:13.643090 1206 raft_consensus.cc:383] T b3b1ea0a825c4d1f8e3edf24cc74d8f7 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:13.645759 1207 ts_tablet_manager.cc:1428] T 268a12df225e4ba59d4a84c2869d4661 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.018s user 0.004s sys 0.002s
I20250114 02:54:13.650861 1207 tablet_bootstrap.cc:492] T 9bfd8fd6d2104fb4a6484cb8c98981bd P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:13.656163 1207 tablet_bootstrap.cc:654] T 9bfd8fd6d2104fb4a6484cb8c98981bd P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:13.662716 1394 ts_tablet_manager.cc:1918] T 61b4b6d4ac364a14a34733b4dd715bb3 P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:13.663107 1394 log.cc:1198] T 61b4b6d4ac364a14a34733b4dd715bb3 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/61b4b6d4ac364a14a34733b4dd715bb3
I20250114 02:54:13.663810 1394 ts_tablet_manager.cc:1939] T 61b4b6d4ac364a14a34733b4dd715bb3 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:13.666332 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet 61b4b6d4ac364a14a34733b4dd715bb3 (table default.Table_0931c459ae794d2091fcc0f51d031548 [id=e279c31bd63b40b9ade03a3b928bf153]) successfully deleted
W20250114 02:54:13.667261 458 catalog_manager.cc:4852] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet 7ecf782bb3d54528aa996abca1ea3a15 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7ecf782bb3d54528aa996abca1ea3a15 already in progress: creating tablet
I20250114 02:54:13.675472 1207 tablet_bootstrap.cc:492] T 9bfd8fd6d2104fb4a6484cb8c98981bd P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:13.675895 1207 ts_tablet_manager.cc:1397] T 9bfd8fd6d2104fb4a6484cb8c98981bd P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.025s user 0.008s sys 0.000s
I20250114 02:54:13.678174 1207 raft_consensus.cc:357] T 9bfd8fd6d2104fb4a6484cb8c98981bd P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:13.678828 1207 raft_consensus.cc:383] T 9bfd8fd6d2104fb4a6484cb8c98981bd P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:13.679076 1207 raft_consensus.cc:738] T 9bfd8fd6d2104fb4a6484cb8c98981bd P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:13.643397 1206 raft_consensus.cc:738] T b3b1ea0a825c4d1f8e3edf24cc74d8f7 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:13.684774 1207 consensus_queue.cc:260] T 9bfd8fd6d2104fb4a6484cb8c98981bd P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:13.685588 471 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_57e1a7bbbda14363912be7c5f5cdb508" } modify_external_catalogs: true
I20250114 02:54:13.686065 471 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_57e1a7bbbda14363912be7c5f5cdb508" } modify_external_catalogs: true
I20250114 02:54:13.687523 1206 consensus_queue.cc:260] T b3b1ea0a825c4d1f8e3edf24cc74d8f7 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:13.693578 1206 ts_tablet_manager.cc:1428] T b3b1ea0a825c4d1f8e3edf24cc74d8f7 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.075s user 0.005s sys 0.000s
I20250114 02:54:13.693910 1207 ts_tablet_manager.cc:1428] T 9bfd8fd6d2104fb4a6484cb8c98981bd P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.018s user 0.006s sys 0.001s
I20250114 02:54:13.696156 1207 tablet_bootstrap.cc:492] T 9589afd0094545c1801b7a0b5eb8698f P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:13.698925 1386 raft_consensus.cc:491] T 276942e5e54246c79f0d4f08297f58ea P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:13.699309 1386 raft_consensus.cc:513] T 276942e5e54246c79f0d4f08297f58ea P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:13.701102 1386 leader_election.cc:290] T 276942e5e54246c79f0d4f08297f58ea P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:13.702503 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "276942e5e54246c79f0d4f08297f58ea" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:13.703090 818 raft_consensus.cc:2463] T 276942e5e54246c79f0d4f08297f58ea P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 0.
I20250114 02:54:13.705492 1042 leader_election.cc:304] T 276942e5e54246c79f0d4f08297f58ea P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:13.706305 1386 raft_consensus.cc:2798] T 276942e5e54246c79f0d4f08297f58ea P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:13.706636 1386 raft_consensus.cc:491] T 276942e5e54246c79f0d4f08297f58ea P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:13.706972 1386 raft_consensus.cc:3054] T 276942e5e54246c79f0d4f08297f58ea P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:13.711951 469 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_322900c20b6647489b2f2a4deecb7879" } new_table_name: "default.Table_569df7e8f022465c96cd2a326e905c72" modify_external_catalogs: true
I20250114 02:54:13.715474 1389 raft_consensus.cc:491] T 4bb9b429af92446bbe13894d5dfe4c47 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:13.715958 1389 raft_consensus.cc:513] T 4bb9b429af92446bbe13894d5dfe4c47 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:13.717844 1389 leader_election.cc:290] T 4bb9b429af92446bbe13894d5dfe4c47 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:13.723105 1206 tablet_bootstrap.cc:492] T 268a12df225e4ba59d4a84c2869d4661 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:13.726133 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bb9b429af92446bbe13894d5dfe4c47" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:13.726001 464 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_123095518d164d3db7fe3bd50af9dbd4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:13.726778 1106 raft_consensus.cc:2463] T 4bb9b429af92446bbe13894d5dfe4c47 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 0.
W20250114 02:54:13.727475 464 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_123095518d164d3db7fe3bd50af9dbd4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:13.728277 753 leader_election.cc:304] T 4bb9b429af92446bbe13894d5dfe4c47 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:13.729179 1389 raft_consensus.cc:2798] T 4bb9b429af92446bbe13894d5dfe4c47 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:13.729485 1389 raft_consensus.cc:491] T 4bb9b429af92446bbe13894d5dfe4c47 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:13.729770 1389 raft_consensus.cc:3054] T 4bb9b429af92446bbe13894d5dfe4c47 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:13.740983 1386 raft_consensus.cc:513] T 276942e5e54246c79f0d4f08297f58ea P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:13.743647 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "276942e5e54246c79f0d4f08297f58ea" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:13.744246 818 raft_consensus.cc:3054] T 276942e5e54246c79f0d4f08297f58ea P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:13.746922 463 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet b63dd90f60a04465b948171e74888e79 (table default.Table_1097f09fb10e441fb639f8d7aa12d487 [id=d8c9691818d042d7b49475fa009c4ca1]). Expected version 3 got 0
I20250114 02:54:13.750240 1086 tablet_service.cc:1467] Processing CreateTablet for tablet fcb58a05f980460884ed81a30a5d59c2 (DEFAULT_TABLE table=default.Table_f290e54e05c7429c9d991c02cfdfea48 [id=fa83160975a0470bb812f677d6ee2a22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:13.750623 794 tablet_service.cc:1467] Processing CreateTablet for tablet fcb58a05f980460884ed81a30a5d59c2 (DEFAULT_TABLE table=default.Table_f290e54e05c7429c9d991c02cfdfea48 [id=fa83160975a0470bb812f677d6ee2a22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:13.751253 1086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcb58a05f980460884ed81a30a5d59c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:13.751597 794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcb58a05f980460884ed81a30a5d59c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:13.752456 793 tablet_service.cc:1467] Processing CreateTablet for tablet ba215eb8422d4f879dd0bf5996543962 (DEFAULT_TABLE table=default.Table_bacdf22532c84100a69a38cfa0ad07e5 [id=86304f6b48fd421cb953070ae7c9b0a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:13.753405 793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba215eb8422d4f879dd0bf5996543962. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:13.753399 1081 tablet_service.cc:1467] Processing CreateTablet for tablet ba215eb8422d4f879dd0bf5996543962 (DEFAULT_TABLE table=default.Table_bacdf22532c84100a69a38cfa0ad07e5 [id=86304f6b48fd421cb953070ae7c9b0a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:13.754410 1081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba215eb8422d4f879dd0bf5996543962. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:13.756731 467 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_b8b1e6d9bd0742f7880a7f55e986d0ee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:13.758167 467 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b8b1e6d9bd0742f7880a7f55e986d0ee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:13.758550 465 catalog_manager.cc:5526] T 8b4bc535c42e43889401f14b6d59659e P 525198593c8e47afaee54e74267b87e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 525198593c8e47afaee54e74267b87e3 (127.0.107.66). New cstate: current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } }
I20250114 02:54:13.759748 465 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 8b4bc535c42e43889401f14b6d59659e (table default.Table_d90b2995b13b4a8fb7e50cc4b79cbf84 [id=f8e40e0aa44b4157ac3438d11685b2e7]). Expected version 1 got 0
I20250114 02:54:13.760483 465 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 4bb9b429af92446bbe13894d5dfe4c47 (table default.Table_acb5bf9354be440bbf6034cf1c535dab [id=22b7edcf4b7e4c31b3c23dc44ccf028e]). Expected version 1 got 0
I20250114 02:54:13.765287 1389 raft_consensus.cc:513] T 4bb9b429af92446bbe13894d5dfe4c47 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:13.767484 792 tablet_service.cc:1467] Processing CreateTablet for tablet aad96f2507974023aaa988e6950feb7a (DEFAULT_TABLE table=default.Table_8344ae9b78e64a038ba7e64f1f807946 [id=f38063ebf94d4084ab0174428503dc29]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:13.767963 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bb9b429af92446bbe13894d5dfe4c47" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:13.768584 792 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aad96f2507974023aaa988e6950feb7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:13.768527 1106 raft_consensus.cc:3054] T 4bb9b429af92446bbe13894d5dfe4c47 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:13.772176 798 tablet_service.cc:1467] Processing CreateTablet for tablet 99b8acb1fe594bf3825c7b9121018db6 (DEFAULT_TABLE table=default.Table_661ed49bcbbb476aa68742b72604d95f [id=00383db95e10461a83394c5bd94ca94a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:13.773056 798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99b8acb1fe594bf3825c7b9121018db6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:13.776865 1085 tablet_service.cc:1467] Processing CreateTablet for tablet 99b8acb1fe594bf3825c7b9121018db6 (DEFAULT_TABLE table=default.Table_661ed49bcbbb476aa68742b72604d95f [id=00383db95e10461a83394c5bd94ca94a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:13.777763 1085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99b8acb1fe594bf3825c7b9121018db6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:13.777673 797 tablet_service.cc:1467] Processing CreateTablet for tablet f464eabfbc0945fa8a5583f00c1dab40 (DEFAULT_TABLE table=default.Table_4fd891b63924429085d78f78cb177f0e [id=d36c71e7562b4ff1bc8be5cf2955000e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:13.778468 797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f464eabfbc0945fa8a5583f00c1dab40. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:13.776865 1083 tablet_service.cc:1467] Processing CreateTablet for tablet aad96f2507974023aaa988e6950feb7a (DEFAULT_TABLE table=default.Table_8344ae9b78e64a038ba7e64f1f807946 [id=f38063ebf94d4084ab0174428503dc29]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:13.782866 1083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aad96f2507974023aaa988e6950feb7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:13.790719 1084 tablet_service.cc:1467] Processing CreateTablet for tablet f464eabfbc0945fa8a5583f00c1dab40 (DEFAULT_TABLE table=default.Table_4fd891b63924429085d78f78cb177f0e [id=d36c71e7562b4ff1bc8be5cf2955000e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:13.791735 1084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f464eabfbc0945fa8a5583f00c1dab40. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:13.794813 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet 92b8d48880434ab99eec7f1e452931fa (table default.Table_3497e36f166142c3a1ef161324c8f45f [id=a3a41d4e0b004a4aaefc1c1009a8f244]) successfully deleted
W20250114 02:54:13.795562 456 catalog_manager.cc:4839] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet 0752ab4b7031402db4e6be8b61cf364f because the tablet was not found. No further retry: Not found: Tablet not found: 0752ab4b7031402db4e6be8b61cf364f
I20250114 02:54:13.801337 940 tablet_service.cc:1467] Processing CreateTablet for tablet fcb58a05f980460884ed81a30a5d59c2 (DEFAULT_TABLE table=default.Table_f290e54e05c7429c9d991c02cfdfea48 [id=fa83160975a0470bb812f677d6ee2a22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:13.802282 937 tablet_service.cc:1467] Processing CreateTablet for tablet 99b8acb1fe594bf3825c7b9121018db6 (DEFAULT_TABLE table=default.Table_661ed49bcbbb476aa68742b72604d95f [id=00383db95e10461a83394c5bd94ca94a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:13.803236 940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcb58a05f980460884ed81a30a5d59c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:13.762197 818 raft_consensus.cc:2463] T 276942e5e54246c79f0d4f08297f58ea P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1.
I20250114 02:54:13.804262 939 tablet_service.cc:1467] Processing CreateTablet for tablet aad96f2507974023aaa988e6950feb7a (DEFAULT_TABLE table=default.Table_8344ae9b78e64a038ba7e64f1f807946 [id=f38063ebf94d4084ab0174428503dc29]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:13.805461 939 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aad96f2507974023aaa988e6950feb7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:13.775547 1106 raft_consensus.cc:2463] T 4bb9b429af92446bbe13894d5dfe4c47 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 1.
I20250114 02:54:13.808396 753 leader_election.cc:304] T 4bb9b429af92446bbe13894d5dfe4c47 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:13.801337 936 tablet_service.cc:1467] Processing CreateTablet for tablet ba215eb8422d4f879dd0bf5996543962 (DEFAULT_TABLE table=default.Table_bacdf22532c84100a69a38cfa0ad07e5 [id=86304f6b48fd421cb953070ae7c9b0a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:13.809381 1294 raft_consensus.cc:2798] T 4bb9b429af92446bbe13894d5dfe4c47 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:13.809868 936 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba215eb8422d4f879dd0bf5996543962. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:13.813198 1042 leader_election.cc:304] T 276942e5e54246c79f0d4f08297f58ea P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:13.814010 937 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99b8acb1fe594bf3825c7b9121018db6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:13.814185 1376 raft_consensus.cc:2798] T 276942e5e54246c79f0d4f08297f58ea P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:13.814682 1376 raft_consensus.cc:695] T 276942e5e54246c79f0d4f08297f58ea P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Becoming Leader. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Running, Role: LEADER
I20250114 02:54:13.815555 1376 consensus_queue.cc:237] T 276942e5e54246c79f0d4f08297f58ea P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:13.821310 941 tablet_service.cc:1467] Processing CreateTablet for tablet f464eabfbc0945fa8a5583f00c1dab40 (DEFAULT_TABLE table=default.Table_4fd891b63924429085d78f78cb177f0e [id=d36c71e7562b4ff1bc8be5cf2955000e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:13.821084 960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bb9b429af92446bbe13894d5dfe4c47" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:13.822549 941 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f464eabfbc0945fa8a5583f00c1dab40. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:13.822446 960 raft_consensus.cc:2463] T 4bb9b429af92446bbe13894d5dfe4c47 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 0.
I20250114 02:54:13.826210 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "276942e5e54246c79f0d4f08297f58ea" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
W20250114 02:54:13.828608 1040 leader_election.cc:343] T 276942e5e54246c79f0d4f08297f58ea P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:13.833808 1207 tablet_bootstrap.cc:654] T 9589afd0094545c1801b7a0b5eb8698f P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:13.847503 1205 ts_tablet_manager.cc:1428] T 83b5b36a26e74668998a19fa0ddf7188 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.226s user 0.006s sys 0.000s
I20250114 02:54:13.630607 1393 ts_tablet_manager.cc:1905] T 61b4b6d4ac364a14a34733b4dd715bb3 P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:13.853018 1205 tablet_bootstrap.cc:492] T 268a12df225e4ba59d4a84c2869d4661 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:13.854213 1386 leader_election.cc:290] T 276942e5e54246c79f0d4f08297f58ea P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Requested vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:13.854774 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "276942e5e54246c79f0d4f08297f58ea" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
I20250114 02:54:13.855224 1389 leader_election.cc:290] T 4bb9b429af92446bbe13894d5dfe4c47 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Requested vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
W20250114 02:54:13.856305 1040 leader_election.cc:343] T 276942e5e54246c79f0d4f08297f58ea P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:13.862406 1294 raft_consensus.cc:695] T 4bb9b429af92446bbe13894d5dfe4c47 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 LEADER]: Becoming Leader. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Running, Role: LEADER
I20250114 02:54:13.863206 1294 consensus_queue.cc:237] T 4bb9b429af92446bbe13894d5dfe4c47 P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:13.867961 1461 consensus_queue.cc:1035] T b4740f6b3a6741368ba68907ad065279 P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:13.874378 1206 tablet_bootstrap.cc:654] T 268a12df225e4ba59d4a84c2869d4661 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:13.879571 1294 consensus_queue.cc:1035] T b4740f6b3a6741368ba68907ad065279 P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:13.887073 1207 tablet_bootstrap.cc:492] T 9589afd0094545c1801b7a0b5eb8698f P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:13.887465 1207 ts_tablet_manager.cc:1397] T 9589afd0094545c1801b7a0b5eb8698f P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.192s user 0.009s sys 0.000s
I20250114 02:54:13.894371 468 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_9a7ce95817ea4e4fb4bc3d8ccb6f5e8c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:13.895870 468 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9a7ce95817ea4e4fb4bc3d8ccb6f5e8c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:13.913908 471 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 8ab8b133191a46e28e1a8e18ec0e79fd
W20250114 02:54:13.914072 469 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1381e26725174b9c877075cf5444f566 has no consensus state
I20250114 02:54:13.914484 471 master_service.cc:946] ReplaceTablet: received request to replace tablet 268a12df225e4ba59d4a84c2869d4661 from {username='slave'} at 127.0.0.1:33540
I20250114 02:54:13.914626 469 master_service.cc:946] ReplaceTablet: received request to replace tablet ed2d137579b8467da5eed4ea2f619a9a from {username='slave'} at 127.0.0.1:33540
I20250114 02:54:13.924619 1207 raft_consensus.cc:357] T 9589afd0094545c1801b7a0b5eb8698f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:13.931478 1207 raft_consensus.cc:383] T 9589afd0094545c1801b7a0b5eb8698f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:13.931742 1207 raft_consensus.cc:738] T 9589afd0094545c1801b7a0b5eb8698f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:13.934993 1207 consensus_queue.cc:260] T 9589afd0094545c1801b7a0b5eb8698f P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:13.940317 471 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_b9119494def445ddba465df3e4513efe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:13.941740 471 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b9119494def445ddba465df3e4513efe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:13.944128 1294 raft_consensus.cc:491] T 8b4bc535c42e43889401f14b6d59659e P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:13.944293 1207 ts_tablet_manager.cc:1428] T 9589afd0094545c1801b7a0b5eb8698f P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.057s user 0.006s sys 0.000s
I20250114 02:54:13.945101 1207 tablet_bootstrap.cc:492] T c20aca761313498193e1d79c10705f8f P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:13.944597 1294 raft_consensus.cc:513] T 8b4bc535c42e43889401f14b6d59659e P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:13.946260 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bb9b429af92446bbe13894d5dfe4c47" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
I20250114 02:54:13.946488 1294 leader_election.cc:290] T 8b4bc535c42e43889401f14b6d59659e P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:13.946817 962 raft_consensus.cc:3054] T 4bb9b429af92446bbe13894d5dfe4c47 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:13.949626 1205 tablet_bootstrap.cc:654] T 268a12df225e4ba59d4a84c2869d4661 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:13.955524 962 raft_consensus.cc:2463] T 4bb9b429af92446bbe13894d5dfe4c47 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 1.
I20250114 02:54:13.948493 1105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b4bc535c42e43889401f14b6d59659e" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:13.963078 1105 raft_consensus.cc:2388] T 8b4bc535c42e43889401f14b6d59659e P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 819f9e3412e54d44abb11d8b3ca877c2 in current term 1: Already voted for candidate 525198593c8e47afaee54e74267b87e3 in this term.
I20250114 02:54:13.949702 1376 raft_consensus.cc:491] T 9589afd0094545c1801b7a0b5eb8698f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:13.963714 1376 raft_consensus.cc:513] T 9589afd0094545c1801b7a0b5eb8698f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:13.965451 1376 leader_election.cc:290] T 9589afd0094545c1801b7a0b5eb8698f P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:13.967489 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9589afd0094545c1801b7a0b5eb8698f" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
W20250114 02:54:13.968919 1042 leader_election.cc:343] T 9589afd0094545c1801b7a0b5eb8698f P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:13.976837 1206 tablet_bootstrap.cc:492] T 268a12df225e4ba59d4a84c2869d4661 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:13.977246 1206 ts_tablet_manager.cc:1397] T 268a12df225e4ba59d4a84c2869d4661 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.254s user 0.010s sys 0.003s
I20250114 02:54:13.967599 959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9589afd0094545c1801b7a0b5eb8698f" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
W20250114 02:54:13.981431 1040 leader_election.cc:343] T 9589afd0094545c1801b7a0b5eb8698f P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:13.981830 1040 leader_election.cc:304] T 9589afd0094545c1801b7a0b5eb8698f P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90d6c2c58bb54e0eafb365625ac33871; no voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2
I20250114 02:54:13.983069 1376 raft_consensus.cc:2743] T 9589afd0094545c1801b7a0b5eb8698f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:13.988574 818 raft_consensus.cc:3054] T b3b1ea0a825c4d1f8e3edf24cc74d8f7 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:13.988623 1408 consensus_queue.cc:1035] T b3b1ea0a825c4d1f8e3edf24cc74d8f7 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:54:13.990213 959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b4bc535c42e43889401f14b6d59659e" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:13.995816 751 leader_election.cc:304] T 8b4bc535c42e43889401f14b6d59659e P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:13.996729 1459 raft_consensus.cc:3054] T 8b4bc535c42e43889401f14b6d59659e P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:13.998647 1386 raft_consensus.cc:491] T 9bfd8fd6d2104fb4a6484cb8c98981bd P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:13.999060 1386 raft_consensus.cc:513] T 9bfd8fd6d2104fb4a6484cb8c98981bd P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:14.000892 1386 leader_election.cc:290] T 9bfd8fd6d2104fb4a6484cb8c98981bd P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:14.001755 959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bfd8fd6d2104fb4a6484cb8c98981bd" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:14.001883 1206 raft_consensus.cc:357] T 268a12df225e4ba59d4a84c2869d4661 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:14.002557 1206 raft_consensus.cc:383] T 268a12df225e4ba59d4a84c2869d4661 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:14.002642 817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bfd8fd6d2104fb4a6484cb8c98981bd" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:14.002810 1206 raft_consensus.cc:738] T 268a12df225e4ba59d4a84c2869d4661 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
W20250114 02:54:14.003126 1040 leader_election.cc:343] T 9bfd8fd6d2104fb4a6484cb8c98981bd P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
W20250114 02:54:14.003924 1042 leader_election.cc:343] T 9bfd8fd6d2104fb4a6484cb8c98981bd P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:14.004330 1042 leader_election.cc:304] T 9bfd8fd6d2104fb4a6484cb8c98981bd P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90d6c2c58bb54e0eafb365625ac33871; no voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2
I20250114 02:54:14.005451 1386 raft_consensus.cc:2743] T 9bfd8fd6d2104fb4a6484cb8c98981bd P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:14.004451 1206 consensus_queue.cc:260] T 268a12df225e4ba59d4a84c2869d4661 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:14.007470 1207 tablet_bootstrap.cc:654] T c20aca761313498193e1d79c10705f8f P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
W20250114 02:54:14.007323 455 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.124:39105, user_credentials={real_user=slave}} blocked reactor thread for 73495.9us
I20250114 02:54:14.015272 1456 consensus_queue.cc:1035] T b3b1ea0a825c4d1f8e3edf24cc74d8f7 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:54:14.016994 1205 tablet_bootstrap.cc:492] T 268a12df225e4ba59d4a84c2869d4661 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:14.017494 1205 ts_tablet_manager.cc:1397] T 268a12df225e4ba59d4a84c2869d4661 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.165s user 0.012s sys 0.002s
I20250114 02:54:14.018918 1459 raft_consensus.cc:2743] T 8b4bc535c42e43889401f14b6d59659e P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:14.019946 1205 raft_consensus.cc:357] T 268a12df225e4ba59d4a84c2869d4661 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:14.020933 1205 raft_consensus.cc:383] T 268a12df225e4ba59d4a84c2869d4661 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:14.021241 1205 raft_consensus.cc:738] T 268a12df225e4ba59d4a84c2869d4661 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:14.021896 1205 consensus_queue.cc:260] T 268a12df225e4ba59d4a84c2869d4661 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:14.024556 1206 ts_tablet_manager.cc:1428] T 268a12df225e4ba59d4a84c2869d4661 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.047s user 0.006s sys 0.001s
I20250114 02:54:14.026314 1206 tablet_bootstrap.cc:492] T b63dd90f60a04465b948171e74888e79 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:14.044112 1205 ts_tablet_manager.cc:1428] T 268a12df225e4ba59d4a84c2869d4661 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.026s user 0.007s sys 0.000s
I20250114 02:54:14.047312 1205 tablet_bootstrap.cc:492] T 276942e5e54246c79f0d4f08297f58ea P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:14.050246 1206 tablet_bootstrap.cc:654] T b63dd90f60a04465b948171e74888e79 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:14.067351 466 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet 290ad07ab8be4791a39194909e000843
I20250114 02:54:14.069965 939 tablet_service.cc:1514] Processing DeleteTablet for tablet 290ad07ab8be4791a39194909e000843 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:13 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:14.065553 1207 tablet_bootstrap.cc:492] T c20aca761313498193e1d79c10705f8f P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:14.073918 793 tablet_service.cc:1514] Processing DeleteTablet for tablet 290ad07ab8be4791a39194909e000843 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:13 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:14.074095 1207 ts_tablet_manager.cc:1397] T c20aca761313498193e1d79c10705f8f P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.129s user 0.013s sys 0.001s
I20250114 02:54:14.077014 1081 tablet_service.cc:1514] Processing DeleteTablet for tablet 290ad07ab8be4791a39194909e000843 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:13 UTC) from {username='slave'} at 127.0.0.1:53876
W20250114 02:54:14.077973 458 catalog_manager.cc:4852] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet 290ad07ab8be4791a39194909e000843 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 290ad07ab8be4791a39194909e000843 already in progress: creating tablet
W20250114 02:54:14.078675 458 catalog_manager.cc:4852] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 290ad07ab8be4791a39194909e000843 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 290ad07ab8be4791a39194909e000843 already in progress: creating tablet
I20250114 02:54:14.084182 1206 tablet_bootstrap.cc:492] T b63dd90f60a04465b948171e74888e79 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:14.084599 1206 ts_tablet_manager.cc:1397] T b63dd90f60a04465b948171e74888e79 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.058s user 0.008s sys 0.001s
I20250114 02:54:14.079711 1207 raft_consensus.cc:357] T c20aca761313498193e1d79c10705f8f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:14.086694 1207 raft_consensus.cc:383] T c20aca761313498193e1d79c10705f8f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:14.086959 1207 raft_consensus.cc:738] T c20aca761313498193e1d79c10705f8f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:14.088618 1207 consensus_queue.cc:260] T c20aca761313498193e1d79c10705f8f P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:14.106137 470 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 8b4bc535c42e43889401f14b6d59659e (table default.Table_d90b2995b13b4a8fb7e50cc4b79cbf84 [id=f8e40e0aa44b4157ac3438d11685b2e7]). Expected version 1 got 0
W20250114 02:54:14.107235 462 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 276942e5e54246c79f0d4f08297f58ea consensus state has no leader
I20250114 02:54:14.107332 1207 ts_tablet_manager.cc:1428] T c20aca761313498193e1d79c10705f8f P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.033s user 0.006s sys 0.001s
I20250114 02:54:14.099907 1206 raft_consensus.cc:357] T b63dd90f60a04465b948171e74888e79 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:14.108067 1206 raft_consensus.cc:383] T b63dd90f60a04465b948171e74888e79 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:14.108350 1206 raft_consensus.cc:738] T b63dd90f60a04465b948171e74888e79 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:14.110003 463 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: default.Table_35a78676616e4b20906b262c9d0c0d9e [id=e984149e9da444f1b2802e3f35a60fd6] alter complete (version 2)
I20250114 02:54:14.119112 467 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_67fb7ad07b314b26b85dd43b95ea1faa" } new_table_name: "default.Table_ec0eefb64458440ebd57a05a7ad0bb9a" modify_external_catalogs: true
I20250114 02:54:14.120115 1206 consensus_queue.cc:260] T b63dd90f60a04465b948171e74888e79 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:14.122745 1206 ts_tablet_manager.cc:1428] T b63dd90f60a04465b948171e74888e79 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.038s user 0.003s sys 0.003s
I20250114 02:54:14.122759 462 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_8a880e0ae5ff4b5d8de8f40f15dd6e02" } modify_external_catalogs: true
I20250114 02:54:14.123534 1206 tablet_bootstrap.cc:492] T 83b5b36a26e74668998a19fa0ddf7188 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:14.123524 462 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_8a880e0ae5ff4b5d8de8f40f15dd6e02" } modify_external_catalogs: true
I20250114 02:54:14.129546 1393 ts_tablet_manager.cc:1918] T 61b4b6d4ac364a14a34733b4dd715bb3 P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:14.130143 1393 log.cc:1198] T 61b4b6d4ac364a14a34733b4dd715bb3 P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/61b4b6d4ac364a14a34733b4dd715bb3
I20250114 02:54:14.131808 1393 ts_tablet_manager.cc:1939] T 61b4b6d4ac364a14a34733b4dd715bb3 P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:14.132117 1206 tablet_bootstrap.cc:654] T 83b5b36a26e74668998a19fa0ddf7188 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:14.144943 1207 tablet_bootstrap.cc:492] T ed2d137579b8467da5eed4ea2f619a9a P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:14.155208 1207 tablet_bootstrap.cc:654] T ed2d137579b8467da5eed4ea2f619a9a P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:14.175740 1206 tablet_bootstrap.cc:492] T 83b5b36a26e74668998a19fa0ddf7188 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:14.176105 1206 ts_tablet_manager.cc:1397] T 83b5b36a26e74668998a19fa0ddf7188 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.053s user 0.015s sys 0.003s
I20250114 02:54:14.179177 1206 raft_consensus.cc:357] T 83b5b36a26e74668998a19fa0ddf7188 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:14.179797 1206 raft_consensus.cc:383] T 83b5b36a26e74668998a19fa0ddf7188 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:14.180100 1206 raft_consensus.cc:738] T 83b5b36a26e74668998a19fa0ddf7188 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:14.180720 1206 consensus_queue.cc:260] T 83b5b36a26e74668998a19fa0ddf7188 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:14.181625 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet 61b4b6d4ac364a14a34733b4dd715bb3 (table default.Table_0931c459ae794d2091fcc0f51d031548 [id=e279c31bd63b40b9ade03a3b928bf153]) successfully deleted
I20250114 02:54:14.182678 1206 ts_tablet_manager.cc:1428] T 83b5b36a26e74668998a19fa0ddf7188 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250114 02:54:14.183439 1206 tablet_bootstrap.cc:492] T 9589afd0094545c1801b7a0b5eb8698f P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:14.196614 1206 tablet_bootstrap.cc:654] T 9589afd0094545c1801b7a0b5eb8698f P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:14.178591 1205 tablet_bootstrap.cc:654] T 276942e5e54246c79f0d4f08297f58ea P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
W20250114 02:54:14.214568 456 catalog_manager.cc:4852] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet 7ecf782bb3d54528aa996abca1ea3a15 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7ecf782bb3d54528aa996abca1ea3a15 already in progress: creating tablet
I20250114 02:54:14.215569 469 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet ed2d137579b8467da5eed4ea2f619a9a
I20250114 02:54:14.216820 469 catalog_manager.cc:6596] ReplaceTablet: tablet ed2d137579b8467da5eed4ea2f619a9a deleted and replaced by tablet 0304df55431e42e1a999ed17dc35e6c0
I20250114 02:54:14.219935 1081 tablet_service.cc:1514] Processing DeleteTablet for tablet ed2d137579b8467da5eed4ea2f619a9a with delete_type TABLET_DATA_DELETED (replaced by tablet 0304df55431e42e1a999ed17dc35e6c0) from {username='slave'} at 127.0.0.1:53876
W20250114 02:54:14.223109 458 catalog_manager.cc:4852] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet ed2d137579b8467da5eed4ea2f619a9a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed2d137579b8467da5eed4ea2f619a9a already in progress: creating tablet
I20250114 02:54:14.226975 471 master_service.cc:946] ReplaceTablet: received request to replace tablet b4740f6b3a6741368ba68907ad065279 from {username='slave'} at 127.0.0.1:33540
I20250114 02:54:14.227470 939 tablet_service.cc:1514] Processing DeleteTablet for tablet ed2d137579b8467da5eed4ea2f619a9a with delete_type TABLET_DATA_DELETED (replaced by tablet 0304df55431e42e1a999ed17dc35e6c0) from {username='slave'} at 127.0.0.1:55068
W20250114 02:54:14.233733 456 catalog_manager.cc:4852] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet 290ad07ab8be4791a39194909e000843 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 290ad07ab8be4791a39194909e000843 already in progress: creating tablet
W20250114 02:54:14.237810 456 catalog_manager.cc:4852] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet ed2d137579b8467da5eed4ea2f619a9a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed2d137579b8467da5eed4ea2f619a9a already in progress: creating tablet
I20250114 02:54:14.242205 1207 tablet_bootstrap.cc:492] T ed2d137579b8467da5eed4ea2f619a9a P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:14.242597 1207 ts_tablet_manager.cc:1397] T ed2d137579b8467da5eed4ea2f619a9a P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.098s user 0.008s sys 0.002s
I20250114 02:54:14.250996 1207 raft_consensus.cc:357] T ed2d137579b8467da5eed4ea2f619a9a P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:14.251672 1207 raft_consensus.cc:383] T ed2d137579b8467da5eed4ea2f619a9a P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:14.251914 1207 raft_consensus.cc:738] T ed2d137579b8467da5eed4ea2f619a9a P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:14.259958 1207 consensus_queue.cc:260] T ed2d137579b8467da5eed4ea2f619a9a P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:14.260643 793 tablet_service.cc:1514] Processing DeleteTablet for tablet ed2d137579b8467da5eed4ea2f619a9a with delete_type TABLET_DATA_DELETED (replaced by tablet 0304df55431e42e1a999ed17dc35e6c0) from {username='slave'} at 127.0.0.1:53078
W20250114 02:54:14.262404 458 catalog_manager.cc:4852] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet ed2d137579b8467da5eed4ea2f619a9a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed2d137579b8467da5eed4ea2f619a9a already in progress: creating tablet
I20250114 02:54:14.263113 1206 tablet_bootstrap.cc:492] T 9589afd0094545c1801b7a0b5eb8698f P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:14.263496 1206 ts_tablet_manager.cc:1397] T 9589afd0094545c1801b7a0b5eb8698f P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.080s user 0.031s sys 0.001s
I20250114 02:54:14.265928 1206 raft_consensus.cc:357] T 9589afd0094545c1801b7a0b5eb8698f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:14.266549 1206 raft_consensus.cc:383] T 9589afd0094545c1801b7a0b5eb8698f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:14.266782 1206 raft_consensus.cc:738] T 9589afd0094545c1801b7a0b5eb8698f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:14.270488 1206 consensus_queue.cc:260] T 9589afd0094545c1801b7a0b5eb8698f P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:14.275307 465 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: default.Table_003a5067105b43fdaba7a58fbd24bc3a [id=5e6aa799d6444bad968f29327ebace9b] alter complete (version 1)
I20250114 02:54:14.276749 1207 ts_tablet_manager.cc:1428] T ed2d137579b8467da5eed4ea2f619a9a P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.034s user 0.007s sys 0.000s
I20250114 02:54:14.277858 1206 ts_tablet_manager.cc:1428] T 9589afd0094545c1801b7a0b5eb8698f P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
W20250114 02:54:14.278076 465 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.66:50663 (request call id 37) took 752 ms (client timeout 1000 ms). Trace:
I20250114 02:54:14.278616 1206 tablet_bootstrap.cc:492] T 179248cb5acf4740b268e0ec8b2852f9 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
W20250114 02:54:14.282878 463 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.67:34825 (request call id 36) took 793 ms (client timeout 1000 ms). Trace:
I20250114 02:54:14.285341 1207 tablet_bootstrap.cc:492] T 179248cb5acf4740b268e0ec8b2852f9 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:14.288527 1459 raft_consensus.cc:491] T b63dd90f60a04465b948171e74888e79 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:14.289122 1459 raft_consensus.cc:513] T b63dd90f60a04465b948171e74888e79 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:14.291047 1207 tablet_bootstrap.cc:654] T 179248cb5acf4740b268e0ec8b2852f9 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:14.291028 1459 leader_election.cc:290] T b63dd90f60a04465b948171e74888e79 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:14.292687 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b63dd90f60a04465b948171e74888e79" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:14.293406 1106 raft_consensus.cc:2388] T b63dd90f60a04465b948171e74888e79 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 819f9e3412e54d44abb11d8b3ca877c2 in current term 1: Already voted for candidate 525198593c8e47afaee54e74267b87e3 in this term.
I20250114 02:54:14.294984 1206 tablet_bootstrap.cc:654] T 179248cb5acf4740b268e0ec8b2852f9 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:14.295326 1081 tablet_service.cc:1514] Processing DeleteTablet for tablet 92b8d48880434ab99eec7f1e452931fa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:13 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:14.295656 1084 tablet_service.cc:1514] Processing DeleteTablet for tablet 83b5b36a26e74668998a19fa0ddf7188 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:12 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:14.296403 1223 tablet_replica.cc:331] T 83b5b36a26e74668998a19fa0ddf7188 P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:14.297080 1223 raft_consensus.cc:2238] T 83b5b36a26e74668998a19fa0ddf7188 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:14.297494 1223 raft_consensus.cc:2267] T 83b5b36a26e74668998a19fa0ddf7188 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:14.297932 468 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_b6330aba5846413c915c50dc4c6e5f11" } modify_external_catalogs: true
I20250114 02:54:14.298475 468 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_b6330aba5846413c915c50dc4c6e5f11" } modify_external_catalogs: true
I20250114 02:54:14.298286 470 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_11c84abe27d74b67bfbd28c249c78b03"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:14.299068 1223 ts_tablet_manager.cc:1905] T 83b5b36a26e74668998a19fa0ddf7188 P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250114 02:54:14.299700 470 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_11c84abe27d74b67bfbd28c249c78b03 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:14.300626 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b63dd90f60a04465b948171e74888e79" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:14.301095 469 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_3f4f2aa33eae44a1ba3c243179bfd590"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:14.301996 751 leader_election.cc:304] T b63dd90f60a04465b948171e74888e79 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
W20250114 02:54:14.302512 469 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3f4f2aa33eae44a1ba3c243179bfd590 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250114 02:54:14.310168 467 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a3fdab330d574c0283ac58bc9e450b87 consensus state has no leader
W20250114 02:54:14.312543 458 catalog_manager.cc:4839] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 92b8d48880434ab99eec7f1e452931fa because the tablet was not found. No further retry: Not found: Tablet not found: 92b8d48880434ab99eec7f1e452931fa
I20250114 02:54:14.313875 467 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_2449658d21e0468f84368f26a2756c00" } new_table_name: "default.Table_3139140b7eec4849ae715c295fbeaa6b" modify_external_catalogs: true
I20250114 02:54:14.316929 1459 raft_consensus.cc:3054] T b63dd90f60a04465b948171e74888e79 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:14.318746 1223 ts_tablet_manager.cc:1918] T 83b5b36a26e74668998a19fa0ddf7188 P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:14.319087 1223 log.cc:1198] T 83b5b36a26e74668998a19fa0ddf7188 P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/83b5b36a26e74668998a19fa0ddf7188
I20250114 02:54:14.319700 1223 ts_tablet_manager.cc:1939] T 83b5b36a26e74668998a19fa0ddf7188 P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
I20250114 02:54:14.322257 1207 tablet_bootstrap.cc:492] T 179248cb5acf4740b268e0ec8b2852f9 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:14.322639 1207 ts_tablet_manager.cc:1397] T 179248cb5acf4740b268e0ec8b2852f9 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.045s user 0.013s sys 0.006s
I20250114 02:54:14.324752 1207 raft_consensus.cc:357] T 179248cb5acf4740b268e0ec8b2852f9 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:14.325382 1207 raft_consensus.cc:383] T 179248cb5acf4740b268e0ec8b2852f9 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:14.325615 1207 raft_consensus.cc:738] T 179248cb5acf4740b268e0ec8b2852f9 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
W20250114 02:54:14.283238 463 rpcz_store.cc:269] 0114 02:54:13.489787 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:13.489915 (+ 128us) service_pool.cc:225] Handling call
0114 02:54:14.282814 (+792899us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:13.978822 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:13.979183 (+ 361us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:13.979204 (+ 21us) write_op.cc:435] Acquired schema lock
0114 02:54:13.979219 (+ 15us) tablet.cc:659] Decoding operations
0114 02:54:13.984253 (+ 5034us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:13.984292 (+ 39us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:13.984314 (+ 22us) tablet.cc:682] Acquiring locks for 1 operations
0114 02:54:13.984468 (+ 154us) tablet.cc:698] Row locks acquired
0114 02:54:13.984480 (+ 12us) write_op.cc:260] PREPARE: finished
0114 02:54:13.984585 (+ 105us) write_op.cc:270] Start()
0114 02:54:13.984664 (+ 79us) write_op.cc:276] Timestamp: P: 1736823253984536 usec, L: 0
0114 02:54:13.984681 (+ 17us) op_driver.cc:348] REPLICATION: starting
0114 02:54:13.985139 (+ 458us) log.cc:844] Serialized 356 byte log entry
0114 02:54:14.102093 (+116954us) op_driver.cc:464] REPLICATION: finished
0114 02:54:14.104338 (+ 2245us) write_op.cc:301] APPLY: starting
0114 02:54:14.104389 (+ 51us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:14.104471 (+ 82us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:14.104575 (+ 104us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:14.104721 (+ 146us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:14.104840 (+ 119us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:14.104877 (+ 37us) write_op.cc:312] APPLY: finished
0114 02:54:14.105062 (+ 185us) log.cc:844] Serialized 23 byte log entry
0114 02:54:14.105192 (+ 130us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:14.105246 (+ 54us) write_op.cc:454] Released schema lock
0114 02:54:14.105273 (+ 27us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:14.105494 (+ 221us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:14.105513 (+ 19us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":6,"child_traces":[["op",{"apply.queue_time_us":2064,"apply.run_cpu_time_us":1142,"apply.run_wall_time_us":1142,"num_ops":1,"prepare.queue_time_us":209467,"prepare.run_cpu_time_us":1816,"prepare.run_wall_time_us":6538,"replication_time_us":117326}]]}
I20250114 02:54:14.326239 1207 consensus_queue.cc:260] T 179248cb5acf4740b268e0ec8b2852f9 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:14.329630 1207 ts_tablet_manager.cc:1428] T 179248cb5acf4740b268e0ec8b2852f9 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20250114 02:54:14.330396 1207 tablet_bootstrap.cc:492] T 25e85b0c5c9a4a3ca35a001359f21167 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:14.340260 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet 83b5b36a26e74668998a19fa0ddf7188 (table default.Table_acca64d3328d47578d6d1ebd594babbd [id=bb08f9cb85c44d108156675ff51b37c1]) successfully deleted
I20250114 02:54:14.340384 939 tablet_service.cc:1514] Processing DeleteTablet for tablet 92b8d48880434ab99eec7f1e452931fa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:13 UTC) from {username='slave'} at 127.0.0.1:55068
W20250114 02:54:14.278386 465 rpcz_store.cc:269] 0114 02:54:13.525302 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:13.530531 (+ 5229us) service_pool.cc:225] Handling call
0114 02:54:14.278020 (+747489us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:13.963021 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:13.963352 (+ 331us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:13.963374 (+ 22us) write_op.cc:435] Acquired schema lock
0114 02:54:13.963386 (+ 12us) tablet.cc:659] Decoding operations
0114 02:54:13.963624 (+ 238us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:13.963657 (+ 33us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:13.963676 (+ 19us) tablet.cc:682] Acquiring locks for 1 operations
0114 02:54:13.963815 (+ 139us) tablet.cc:698] Row locks acquired
0114 02:54:13.963829 (+ 14us) write_op.cc:260] PREPARE: finished
0114 02:54:13.963931 (+ 102us) write_op.cc:270] Start()
0114 02:54:13.964014 (+ 83us) write_op.cc:276] Timestamp: P: 1736823253963884 usec, L: 0
0114 02:54:13.964029 (+ 15us) op_driver.cc:348] REPLICATION: starting
0114 02:54:13.964486 (+ 457us) log.cc:844] Serialized 530 byte log entry
0114 02:54:14.041455 (+ 76969us) op_driver.cc:464] REPLICATION: finished
0114 02:54:14.054588 (+ 13133us) write_op.cc:301] APPLY: starting
0114 02:54:14.054645 (+ 57us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:14.054721 (+ 76us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:14.054826 (+ 105us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:14.054972 (+ 146us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:14.055113 (+ 141us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:14.055148 (+ 35us) write_op.cc:312] APPLY: finished
0114 02:54:14.055310 (+ 162us) log.cc:844] Serialized 23 byte log entry
0114 02:54:14.055445 (+ 135us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:14.079962 (+ 24517us) spinlock_profiling.cc:243] Waited 4.09 ms on lock 0x7b0c0009f6f0. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f6461ae1a81 00007f6461c214c0 00007f6461c21d90 00007f6461c21c0e 00007f6461bc4510 00007f6461bc3718 00007f6461bc311c 00007f6461baf1b9 00007f6461bae983 00007f6461bb2b91 00007f6461bb2b49
0114 02:54:14.080019 (+ 57us) write_op.cc:454] Released schema lock
0114 02:54:14.080052 (+ 33us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:14.116542 (+ 36490us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:14.116573 (+ 31us) write_op.cc:454] Released schema lock
Related trace 'op':
0114 02:54:14.208704 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:14.209052 (+ 348us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:14.209073 (+ 21us) write_op.cc:435] Acquired schema lock
0114 02:54:14.209086 (+ 13us) tablet.cc:659] Decoding operations
0114 02:54:14.209339 (+ 253us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:14.209371 (+ 32us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:14.209392 (+ 21us) tablet.cc:682] Acquiring locks for 1 operations
0114 02:54:14.209520 (+ 128us) tablet.cc:698] Row locks acquired
0114 02:54:14.209533 (+ 13us) write_op.cc:260] PREPARE: finished
0114 02:54:14.209630 (+ 97us) write_op.cc:270] Start()
0114 02:54:14.209724 (+ 94us) write_op.cc:276] Timestamp: P: 1736823254209560 usec, L: 0
0114 02:54:14.209746 (+ 22us) op_driver.cc:348] REPLICATION: starting
0114 02:54:14.210355 (+ 609us) log.cc:844] Serialized 356 byte log entry
0114 02:54:14.272033 (+ 61678us) op_driver.cc:464] REPLICATION: finished
0114 02:54:14.272314 (+ 281us) write_op.cc:301] APPLY: starting
0114 02:54:14.272367 (+ 53us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:14.272469 (+ 102us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:14.272570 (+ 101us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:14.272709 (+ 139us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:14.272830 (+ 121us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:14.272889 (+ 59us) write_op.cc:312] APPLY: finished
0114 02:54:14.273067 (+ 178us) log.cc:844] Serialized 23 byte log entry
0114 02:54:14.273204 (+ 137us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:14.273260 (+ 56us) write_op.cc:454] Released schema lock
0114 02:54:14.273286 (+ 26us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":5,"child_traces":[["op",{"apply.queue_time_us":12978,"apply.run_cpu_time_us":1930,"apply.run_wall_time_us":61872,"num_ops":1,"prepare.queue_time_us":200817,"prepare.run_cpu_time_us":1660,"prepare.run_wall_time_us":1671,"replication_time_us":77336,"spinlock_wait_cycles":9396096,"thread_start_us":13608,"threads_started":1}],["op",{"apply.queue_time_us":98,"apply.run_cpu_time_us":1133,"apply.run_wall_time_us":1133,"num_ops":1,"prepare.queue_time_us":108777,"prepare.run_cpu_time_us":1881,"prepare.run_wall_time_us":17670,"replication_time_us":62118}]]}
I20250114 02:54:14.351354 1207 tablet_bootstrap.cc:654] T 25e85b0c5c9a4a3ca35a001359f21167 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:14.366492 1207 tablet_bootstrap.cc:492] T 25e85b0c5c9a4a3ca35a001359f21167 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:14.366881 1207 ts_tablet_manager.cc:1397] T 25e85b0c5c9a4a3ca35a001359f21167 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.037s user 0.008s sys 0.012s
I20250114 02:54:14.369215 1207 raft_consensus.cc:357] T 25e85b0c5c9a4a3ca35a001359f21167 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:14.369836 1207 raft_consensus.cc:383] T 25e85b0c5c9a4a3ca35a001359f21167 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:14.370116 1207 raft_consensus.cc:738] T 25e85b0c5c9a4a3ca35a001359f21167 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:14.370687 1207 consensus_queue.cc:260] T 25e85b0c5c9a4a3ca35a001359f21167 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:14.378684 1294 raft_consensus.cc:491] T 9589afd0094545c1801b7a0b5eb8698f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 02:54:14.379331 456 catalog_manager.cc:4839] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet 92b8d48880434ab99eec7f1e452931fa because the tablet was not found. No further retry: Not found: Tablet not found: 92b8d48880434ab99eec7f1e452931fa
I20250114 02:54:14.379168 1294 raft_consensus.cc:513] T 9589afd0094545c1801b7a0b5eb8698f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:14.379964 1205 tablet_bootstrap.cc:492] T 276942e5e54246c79f0d4f08297f58ea P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:14.380409 1205 ts_tablet_manager.cc:1397] T 276942e5e54246c79f0d4f08297f58ea P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.333s user 0.010s sys 0.000s
I20250114 02:54:14.381433 1294 leader_election.cc:290] T 9589afd0094545c1801b7a0b5eb8698f P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:14.382622 1205 raft_consensus.cc:357] T 276942e5e54246c79f0d4f08297f58ea P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:14.382794 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9589afd0094545c1801b7a0b5eb8698f" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:14.383257 1207 ts_tablet_manager.cc:1428] T 25e85b0c5c9a4a3ca35a001359f21167 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.016s user 0.006s sys 0.001s
I20250114 02:54:14.383255 1205 raft_consensus.cc:383] T 276942e5e54246c79f0d4f08297f58ea P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:14.383384 1106 raft_consensus.cc:2463] T 9589afd0094545c1801b7a0b5eb8698f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 0.
I20250114 02:54:14.383503 1205 raft_consensus.cc:738] T 276942e5e54246c79f0d4f08297f58ea P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:14.384063 1205 consensus_queue.cc:260] T 276942e5e54246c79f0d4f08297f58ea P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:14.386060 1205 ts_tablet_manager.cc:1428] T 276942e5e54246c79f0d4f08297f58ea P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250114 02:54:14.391780 753 leader_election.cc:304] T 9589afd0094545c1801b7a0b5eb8698f P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:14.393453 1460 raft_consensus.cc:2798] T 9589afd0094545c1801b7a0b5eb8698f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:14.393771 1460 raft_consensus.cc:491] T 9589afd0094545c1801b7a0b5eb8698f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:14.394060 1460 raft_consensus.cc:3054] T 9589afd0094545c1801b7a0b5eb8698f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:14.398300 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9589afd0094545c1801b7a0b5eb8698f" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:14.399605 1207 tablet_bootstrap.cc:492] T 96b06fe4173343dabd726d253b9f9b5f P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
W20250114 02:54:14.399678 751 leader_election.cc:343] T 9589afd0094545c1801b7a0b5eb8698f P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:14.401290 793 tablet_service.cc:1514] Processing DeleteTablet for tablet 61b4b6d4ac364a14a34733b4dd715bb3 with delete_type TABLET_DATA_DELETED (replaced by tablet 6448eea21ab2422c97da2494658ded3f) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:14.402926 1205 tablet_bootstrap.cc:492] T ed2d137579b8467da5eed4ea2f619a9a P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
W20250114 02:54:14.402843 458 catalog_manager.cc:4839] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet 61b4b6d4ac364a14a34733b4dd715bb3 because the tablet was not found. No further retry: Not found: Tablet not found: 61b4b6d4ac364a14a34733b4dd715bb3
I20250114 02:54:14.406363 1459 raft_consensus.cc:2743] T b63dd90f60a04465b948171e74888e79 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:14.403563 1460 raft_consensus.cc:513] T 9589afd0094545c1801b7a0b5eb8698f P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:14.409309 1207 tablet_bootstrap.cc:654] T 96b06fe4173343dabd726d253b9f9b5f P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:14.413970 1205 tablet_bootstrap.cc:654] T ed2d137579b8467da5eed4ea2f619a9a P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:14.413972 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9589afd0094545c1801b7a0b5eb8698f" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:14.414534 1106 raft_consensus.cc:3054] T 9589afd0094545c1801b7a0b5eb8698f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:14.416965 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9589afd0094545c1801b7a0b5eb8698f" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
W20250114 02:54:14.418272 751 leader_election.cc:343] T 9589afd0094545c1801b7a0b5eb8698f P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:14.420892 1106 raft_consensus.cc:2463] T 9589afd0094545c1801b7a0b5eb8698f P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 1.
I20250114 02:54:14.422092 753 leader_election.cc:304] T 9589afd0094545c1801b7a0b5eb8698f P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters: 525198593c8e47afaee54e74267b87e3
I20250114 02:54:14.423082 1294 raft_consensus.cc:2798] T 9589afd0094545c1801b7a0b5eb8698f P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:14.430622 1206 tablet_bootstrap.cc:492] T 179248cb5acf4740b268e0ec8b2852f9 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:14.430899 1205 tablet_bootstrap.cc:492] T ed2d137579b8467da5eed4ea2f619a9a P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:14.431020 1206 ts_tablet_manager.cc:1397] T 179248cb5acf4740b268e0ec8b2852f9 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.153s user 0.010s sys 0.020s
I20250114 02:54:14.431370 1205 ts_tablet_manager.cc:1397] T ed2d137579b8467da5eed4ea2f619a9a P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.009s
I20250114 02:54:14.432776 1459 raft_consensus.cc:491] T 268a12df225e4ba59d4a84c2869d4661 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:14.433204 1205 raft_consensus.cc:357] T ed2d137579b8467da5eed4ea2f619a9a P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:14.433208 1459 raft_consensus.cc:513] T 268a12df225e4ba59d4a84c2869d4661 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:14.433662 1205 raft_consensus.cc:383] T ed2d137579b8467da5eed4ea2f619a9a P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:14.433810 1205 raft_consensus.cc:738] T ed2d137579b8467da5eed4ea2f619a9a P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:14.434286 1205 consensus_queue.cc:260] T ed2d137579b8467da5eed4ea2f619a9a P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:14.436506 465 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_ca7bc63288b64f51806ca0f2a9129ff6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:14.437376 1207 tablet_bootstrap.cc:492] T 96b06fe4173343dabd726d253b9f9b5f P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:14.437779 1207 ts_tablet_manager.cc:1397] T 96b06fe4173343dabd726d253b9f9b5f P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.038s user 0.015s sys 0.007s
W20250114 02:54:14.437985 465 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ca7bc63288b64f51806ca0f2a9129ff6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:14.440044 1207 raft_consensus.cc:357] T 96b06fe4173343dabd726d253b9f9b5f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:14.440737 1207 raft_consensus.cc:383] T 96b06fe4173343dabd726d253b9f9b5f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:14.435295 1460 leader_election.cc:290] T 9589afd0094545c1801b7a0b5eb8698f P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Requested vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:14.440997 1207 raft_consensus.cc:738] T 96b06fe4173343dabd726d253b9f9b5f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:14.444504 1459 leader_election.cc:290] T 268a12df225e4ba59d4a84c2869d4661 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:14.445562 1294 raft_consensus.cc:695] T 9589afd0094545c1801b7a0b5eb8698f P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 LEADER]: Becoming Leader. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Running, Role: LEADER
I20250114 02:54:14.446971 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "268a12df225e4ba59d4a84c2869d4661" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:14.446861 1294 consensus_queue.cc:237] T 9589afd0094545c1801b7a0b5eb8698f P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:14.447623 962 raft_consensus.cc:2463] T 268a12df225e4ba59d4a84c2869d4661 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 0.
I20250114 02:54:14.450158 1206 raft_consensus.cc:357] T 179248cb5acf4740b268e0ec8b2852f9 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:14.450779 1206 raft_consensus.cc:383] T 179248cb5acf4740b268e0ec8b2852f9 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:14.451012 1206 raft_consensus.cc:738] T 179248cb5acf4740b268e0ec8b2852f9 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:14.451598 1206 consensus_queue.cc:260] T 179248cb5acf4740b268e0ec8b2852f9 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:14.453241 1205 ts_tablet_manager.cc:1428] T ed2d137579b8467da5eed4ea2f619a9a P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.022s user 0.005s sys 0.001s
I20250114 02:54:14.453747 751 leader_election.cc:304] T 268a12df225e4ba59d4a84c2869d4661 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
I20250114 02:54:14.454144 1205 tablet_bootstrap.cc:492] T 25e85b0c5c9a4a3ca35a001359f21167 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:14.458498 1206 ts_tablet_manager.cc:1428] T 179248cb5acf4740b268e0ec8b2852f9 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.027s user 0.006s sys 0.000s
I20250114 02:54:14.441551 1207 consensus_queue.cc:260] T 96b06fe4173343dabd726d253b9f9b5f P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:14.459297 1206 tablet_bootstrap.cc:492] T 25e85b0c5c9a4a3ca35a001359f21167 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:14.461124 1207 ts_tablet_manager.cc:1428] T 96b06fe4173343dabd726d253b9f9b5f P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.023s user 0.007s sys 0.000s
I20250114 02:54:14.464759 1205 tablet_bootstrap.cc:654] T 25e85b0c5c9a4a3ca35a001359f21167 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:14.466454 1207 tablet_bootstrap.cc:492] T a3fdab330d574c0283ac58bc9e450b87 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:14.467514 1294 raft_consensus.cc:2798] T 268a12df225e4ba59d4a84c2869d4661 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:14.467893 1294 raft_consensus.cc:491] T 268a12df225e4ba59d4a84c2869d4661 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:14.468194 1294 raft_consensus.cc:3054] T 268a12df225e4ba59d4a84c2869d4661 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:14.471812 1207 tablet_bootstrap.cc:654] T a3fdab330d574c0283ac58bc9e450b87 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
W20250114 02:54:14.479080 455 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet b63dd90f60a04465b948171e74888e79 (table default.Table_1097f09fb10e441fb639f8d7aa12d487 [id=d8c9691818d042d7b49475fa009c4ca1], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet b63dd90f60a04465b948171e74888e79 consensus state has no leader
I20250114 02:54:14.483989 1294 raft_consensus.cc:513] T 268a12df225e4ba59d4a84c2869d4661 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:14.495318 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "268a12df225e4ba59d4a84c2869d4661" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:14.496527 1105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "268a12df225e4ba59d4a84c2869d4661" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:14.496867 1235 consensus_queue.cc:1035] T 8b4bc535c42e43889401f14b6d59659e P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:14.495947 1106 raft_consensus.cc:2463] T 268a12df225e4ba59d4a84c2869d4661 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 0.
I20250114 02:54:14.499337 471 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_d50b0a8b26f644898efcb62a0d72fd1b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:14.500378 1206 tablet_bootstrap.cc:654] T 25e85b0c5c9a4a3ca35a001359f21167 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
W20250114 02:54:14.500861 471 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d50b0a8b26f644898efcb62a0d72fd1b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:14.501685 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "268a12df225e4ba59d4a84c2869d4661" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
I20250114 02:54:14.502388 962 raft_consensus.cc:3054] T 268a12df225e4ba59d4a84c2869d4661 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:14.502673 1294 leader_election.cc:290] T 268a12df225e4ba59d4a84c2869d4661 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Requested vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
W20250114 02:54:14.511090 1423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 02:54:14.511894 1423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 02:54:14.512650 1423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250114 02:54:14.514577 1084 tablet_service.cc:1467] Processing CreateTablet for tablet 55b68586ef8b45439a7dd8dcd2efcd79 (DEFAULT_TABLE table=default.Table_42c4106f7e6644d79bb958e314e449e5 [id=c73ede1a5f664d8eb204fb9a012f8f7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:14.515126 1081 tablet_service.cc:1467] Processing CreateTablet for tablet 321b9ad6d1ac4d1c80c7a01faf2015af (DEFAULT_TABLE table=default.Table_efc3b170c00f4ce09b2601f7f5ebf3b8 [id=dba5ff68f085474caeaf34c387ecac6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:14.515389 1206 tablet_bootstrap.cc:492] T 25e85b0c5c9a4a3ca35a001359f21167 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:14.515537 1084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55b68586ef8b45439a7dd8dcd2efcd79. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:14.515756 1206 ts_tablet_manager.cc:1397] T 25e85b0c5c9a4a3ca35a001359f21167 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.057s user 0.005s sys 0.003s
W20250114 02:54:14.519688 467 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet de4ca83a69704a368a25954cdc1b9392 consensus state has no leader
I20250114 02:54:14.520145 1086 tablet_service.cc:1467] Processing CreateTablet for tablet 26328f8e32bc49ed9b224f0e08b9f949 (DEFAULT_TABLE table=default.Table_393cdd3f26b649d6b7b51051237e2c81 [id=a01ec721fefc41d6bc4009ca0c327fab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:14.520866 467 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_003a5067105b43fdaba7a58fbd24bc3a" } new_table_name: "default.Table_3652257c54c3434da1b93eb0481a0df7" modify_external_catalogs: true
I20250114 02:54:14.523465 1206 raft_consensus.cc:357] T 25e85b0c5c9a4a3ca35a001359f21167 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:14.516084 1081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 321b9ad6d1ac4d1c80c7a01faf2015af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:14.525404 1086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26328f8e32bc49ed9b224f0e08b9f949. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:14.524075 1206 raft_consensus.cc:383] T 25e85b0c5c9a4a3ca35a001359f21167 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:14.526901 1206 raft_consensus.cc:738] T 25e85b0c5c9a4a3ca35a001359f21167 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:14.527027 939 tablet_service.cc:1467] Processing CreateTablet for tablet 55b68586ef8b45439a7dd8dcd2efcd79 (DEFAULT_TABLE table=default.Table_42c4106f7e6644d79bb958e314e449e5 [id=c73ede1a5f664d8eb204fb9a012f8f7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:14.528036 939 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55b68586ef8b45439a7dd8dcd2efcd79. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:14.527598 1206 consensus_queue.cc:260] T 25e85b0c5c9a4a3ca35a001359f21167 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:14.530004 1206 ts_tablet_manager.cc:1428] T 25e85b0c5c9a4a3ca35a001359f21167 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.014s user 0.004s sys 0.002s
I20250114 02:54:14.530551 962 raft_consensus.cc:2463] T 268a12df225e4ba59d4a84c2869d4661 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 1.
I20250114 02:54:14.530812 1206 tablet_bootstrap.cc:492] T c20aca761313498193e1d79c10705f8f P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:14.531687 751 leader_election.cc:304] T 268a12df225e4ba59d4a84c2869d4661 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters: 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:14.532750 1496 raft_consensus.cc:2798] T 268a12df225e4ba59d4a84c2869d4661 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:14.533352 936 tablet_service.cc:1467] Processing CreateTablet for tablet 321b9ad6d1ac4d1c80c7a01faf2015af (DEFAULT_TABLE table=default.Table_efc3b170c00f4ce09b2601f7f5ebf3b8 [id=dba5ff68f085474caeaf34c387ecac6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:14.533730 940 tablet_service.cc:1467] Processing CreateTablet for tablet 26328f8e32bc49ed9b224f0e08b9f949 (DEFAULT_TABLE table=default.Table_393cdd3f26b649d6b7b51051237e2c81 [id=a01ec721fefc41d6bc4009ca0c327fab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:14.534638 936 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 321b9ad6d1ac4d1c80c7a01faf2015af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:14.535516 1496 raft_consensus.cc:695] T 268a12df225e4ba59d4a84c2869d4661 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 LEADER]: Becoming Leader. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Running, Role: LEADER
I20250114 02:54:14.512154 466 catalog_manager.cc:5526] T 4bb9b429af92446bbe13894d5dfe4c47 P 819f9e3412e54d44abb11d8b3ca877c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65). New cstate: current_term: 1 leader_uuid: "819f9e3412e54d44abb11d8b3ca877c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:14.536692 1386 raft_consensus.cc:491] T ed2d137579b8467da5eed4ea2f619a9a P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:14.536574 1496 consensus_queue.cc:237] T 268a12df225e4ba59d4a84c2869d4661 P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:14.536844 466 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 4bb9b429af92446bbe13894d5dfe4c47 (table default.Table_acb5bf9354be440bbf6034cf1c535dab [id=22b7edcf4b7e4c31b3c23dc44ccf028e]). Expected version 1 got 0
I20250114 02:54:14.537159 1386 raft_consensus.cc:513] T ed2d137579b8467da5eed4ea2f619a9a P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:14.538048 940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26328f8e32bc49ed9b224f0e08b9f949. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:14.539005 1386 leader_election.cc:290] T ed2d137579b8467da5eed4ea2f619a9a P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:14.540573 817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed2d137579b8467da5eed4ea2f619a9a" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:14.541174 466 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet b63dd90f60a04465b948171e74888e79 (table default.Table_1097f09fb10e441fb639f8d7aa12d487 [id=d8c9691818d042d7b49475fa009c4ca1]). Expected version 3 got 0
I20250114 02:54:14.541550 1083 tablet_service.cc:1467] Processing CreateTablet for tablet 53a8bb487ad64157ad3baecf5d866eca (DEFAULT_TABLE table=default.Table_5d3b333f1d99463b8ad3b4318c8f29df [id=0f1c410e9c5c40ab8b8fe4e4bf68e8ea]), partition=RANGE (key) PARTITION UNBOUNDED
W20250114 02:54:14.541781 1042 leader_election.cc:343] T ed2d137579b8467da5eed4ea2f619a9a P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:14.542615 1083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53a8bb487ad64157ad3baecf5d866eca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:14.546665 1206 tablet_bootstrap.cc:654] T c20aca761313498193e1d79c10705f8f P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:14.549889 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed2d137579b8467da5eed4ea2f619a9a" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:14.556402 962 raft_consensus.cc:2463] T ed2d137579b8467da5eed4ea2f619a9a P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 0.
I20250114 02:54:14.557482 1040 leader_election.cc:304] T ed2d137579b8467da5eed4ea2f619a9a P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters: 819f9e3412e54d44abb11d8b3ca877c2
I20250114 02:54:14.559448 1386 raft_consensus.cc:2798] T ed2d137579b8467da5eed4ea2f619a9a P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:14.559501 937 tablet_service.cc:1467] Processing CreateTablet for tablet 53a8bb487ad64157ad3baecf5d866eca (DEFAULT_TABLE table=default.Table_5d3b333f1d99463b8ad3b4318c8f29df [id=0f1c410e9c5c40ab8b8fe4e4bf68e8ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:14.559846 1386 raft_consensus.cc:491] T ed2d137579b8467da5eed4ea2f619a9a P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:14.560487 937 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53a8bb487ad64157ad3baecf5d866eca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:14.560145 1386 raft_consensus.cc:3054] T ed2d137579b8467da5eed4ea2f619a9a P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:14.562835 1205 tablet_bootstrap.cc:492] T 25e85b0c5c9a4a3ca35a001359f21167 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:14.564749 793 tablet_service.cc:1467] Processing CreateTablet for tablet 55b68586ef8b45439a7dd8dcd2efcd79 (DEFAULT_TABLE table=default.Table_42c4106f7e6644d79bb958e314e449e5 [id=c73ede1a5f664d8eb204fb9a012f8f7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:14.565861 793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55b68586ef8b45439a7dd8dcd2efcd79. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:14.571461 1085 tablet_service.cc:1467] Processing CreateTablet for tablet 66bfa92db9874750b3c340e31a06c412 (DEFAULT_TABLE table=default.Table_b0cd93e5a498450e84fb0f1ba4be4505 [id=c70751b59bfc4c6fa60d0e8bb40d5f36]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:14.572477 1085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66bfa92db9874750b3c340e31a06c412. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:14.577059 1082 tablet_service.cc:1467] Processing CreateTablet for tablet 1381e26725174b9c877075cf5444f566 (DEFAULT_TABLE table=default.Table_569df7e8f022465c96cd2a326e905c72 [id=d2ea84ea59ad4b03ad16bb3052a441f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:14.578074 1082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1381e26725174b9c877075cf5444f566. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:14.577391 937 tablet_service.cc:1467] Processing CreateTablet for tablet 66bfa92db9874750b3c340e31a06c412 (DEFAULT_TABLE table=default.Table_b0cd93e5a498450e84fb0f1ba4be4505 [id=c70751b59bfc4c6fa60d0e8bb40d5f36]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:14.578042 941 tablet_service.cc:1467] Processing CreateTablet for tablet 1381e26725174b9c877075cf5444f566 (DEFAULT_TABLE table=default.Table_569df7e8f022465c96cd2a326e905c72 [id=d2ea84ea59ad4b03ad16bb3052a441f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:14.578953 941 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1381e26725174b9c877075cf5444f566. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:14.580389 1205 ts_tablet_manager.cc:1397] T 25e85b0c5c9a4a3ca35a001359f21167 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.126s user 0.015s sys 0.005s
I20250114 02:54:14.584563 1235 raft_consensus.cc:491] T 83b5b36a26e74668998a19fa0ddf7188 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:14.584975 1235 raft_consensus.cc:513] T 83b5b36a26e74668998a19fa0ddf7188 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:14.586746 1206 tablet_bootstrap.cc:492] T c20aca761313498193e1d79c10705f8f P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:14.586813 1235 leader_election.cc:290] T 83b5b36a26e74668998a19fa0ddf7188 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:14.587141 1206 ts_tablet_manager.cc:1397] T c20aca761313498193e1d79c10705f8f P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.056s user 0.010s sys 0.000s
I20250114 02:54:14.588917 1386 raft_consensus.cc:513] T ed2d137579b8467da5eed4ea2f619a9a P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
W20250114 02:54:14.589759 898 leader_election.cc:343] T 83b5b36a26e74668998a19fa0ddf7188 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Not found: Tablet not found: 83b5b36a26e74668998a19fa0ddf7188
I20250114 02:54:14.591042 817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83b5b36a26e74668998a19fa0ddf7188" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:14.591215 1386 leader_election.cc:290] T ed2d137579b8467da5eed4ea2f619a9a P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Requested vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:14.591495 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed2d137579b8467da5eed4ea2f619a9a" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
I20250114 02:54:14.591643 817 raft_consensus.cc:2463] T 83b5b36a26e74668998a19fa0ddf7188 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:14.592011 962 raft_consensus.cc:3054] T ed2d137579b8467da5eed4ea2f619a9a P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:14.592170 816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed2d137579b8467da5eed4ea2f619a9a" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
W20250114 02:54:14.593495 1042 leader_election.cc:343] T ed2d137579b8467da5eed4ea2f619a9a P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:14.593780 937 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66bfa92db9874750b3c340e31a06c412. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:14.594537 898 leader_election.cc:304] T 83b5b36a26e74668998a19fa0ddf7188 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters: 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:14.596618 1408 raft_consensus.cc:2798] T 83b5b36a26e74668998a19fa0ddf7188 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:14.596905 1408 raft_consensus.cc:491] T 83b5b36a26e74668998a19fa0ddf7188 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:14.597199 1408 raft_consensus.cc:3054] T 83b5b36a26e74668998a19fa0ddf7188 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:14.598089 962 raft_consensus.cc:2463] T ed2d137579b8467da5eed4ea2f619a9a P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1.
I20250114 02:54:14.599242 1040 leader_election.cc:304] T ed2d137579b8467da5eed4ea2f619a9a P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters: 819f9e3412e54d44abb11d8b3ca877c2
I20250114 02:54:14.601650 1206 raft_consensus.cc:357] T c20aca761313498193e1d79c10705f8f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:14.602293 1206 raft_consensus.cc:383] T c20aca761313498193e1d79c10705f8f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:14.602527 1206 raft_consensus.cc:738] T c20aca761313498193e1d79c10705f8f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:14.604096 1235 consensus_queue.cc:1035] T 8b4bc535c42e43889401f14b6d59659e P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:14.606482 1386 raft_consensus.cc:2798] T ed2d137579b8467da5eed4ea2f619a9a P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:14.607291 1205 raft_consensus.cc:357] T 25e85b0c5c9a4a3ca35a001359f21167 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:14.608116 1205 raft_consensus.cc:383] T 25e85b0c5c9a4a3ca35a001359f21167 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:14.608448 1205 raft_consensus.cc:738] T 25e85b0c5c9a4a3ca35a001359f21167 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:14.609084 1205 consensus_queue.cc:260] T 25e85b0c5c9a4a3ca35a001359f21167 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:14.611611 1205 ts_tablet_manager.cc:1428] T 25e85b0c5c9a4a3ca35a001359f21167 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.031s user 0.005s sys 0.000s
I20250114 02:54:14.618588 1206 consensus_queue.cc:260] T c20aca761313498193e1d79c10705f8f P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:14.620796 1080 tablet_service.cc:1467] Processing CreateTablet for tablet bc46815b4803418c95ea9c039a4fae88 (DEFAULT_TABLE table=default.Table_6a4fe39298514187b65def5841d3a992 [id=5c5a7974ad2f4e58ba5c47a6eeaaac62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:14.622011 1080 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc46815b4803418c95ea9c039a4fae88. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:14.624231 1086 tablet_service.cc:1467] Processing CreateTablet for tablet 266da813b4514329be0ad97d07b7114f (DEFAULT_TABLE table=default.Table_ae220acb7e0345bdbb80b11398e4d153 [id=bcf9e834733d49a8bc235cd303a75a35]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:14.625274 1086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 266da813b4514329be0ad97d07b7114f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:14.626796 936 tablet_service.cc:1467] Processing CreateTablet for tablet 266da813b4514329be0ad97d07b7114f (DEFAULT_TABLE table=default.Table_ae220acb7e0345bdbb80b11398e4d153 [id=bcf9e834733d49a8bc235cd303a75a35]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:14.628530 941 tablet_service.cc:1467] Processing CreateTablet for tablet bc46815b4803418c95ea9c039a4fae88 (DEFAULT_TABLE table=default.Table_6a4fe39298514187b65def5841d3a992 [id=5c5a7974ad2f4e58ba5c47a6eeaaac62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:14.638056 936 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 266da813b4514329be0ad97d07b7114f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:14.639176 962 raft_consensus.cc:3054] T 276942e5e54246c79f0d4f08297f58ea P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:14.640089 1205 tablet_bootstrap.cc:492] T c20aca761313498193e1d79c10705f8f P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:14.644481 941 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc46815b4803418c95ea9c039a4fae88. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:14.645044 792 tablet_service.cc:1467] Processing CreateTablet for tablet 321b9ad6d1ac4d1c80c7a01faf2015af (DEFAULT_TABLE table=default.Table_efc3b170c00f4ce09b2601f7f5ebf3b8 [id=dba5ff68f085474caeaf34c387ecac6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:14.645267 797 tablet_service.cc:1467] Processing CreateTablet for tablet 26328f8e32bc49ed9b224f0e08b9f949 (DEFAULT_TABLE table=default.Table_393cdd3f26b649d6b7b51051237e2c81 [id=a01ec721fefc41d6bc4009ca0c327fab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:14.646179 792 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 321b9ad6d1ac4d1c80c7a01faf2015af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:14.646754 797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26328f8e32bc49ed9b224f0e08b9f949. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:14.654246 1206 ts_tablet_manager.cc:1428] T c20aca761313498193e1d79c10705f8f P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.067s user 0.004s sys 0.003s
I20250114 02:54:14.655042 1206 tablet_bootstrap.cc:492] T ed2d137579b8467da5eed4ea2f619a9a P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:14.656986 1235 raft_consensus.cc:491] T 25e85b0c5c9a4a3ca35a001359f21167 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:14.657375 1235 raft_consensus.cc:513] T 25e85b0c5c9a4a3ca35a001359f21167 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:14.666532 1083 tablet_service.cc:1467] Processing CreateTablet for tablet 6448eea21ab2422c97da2494658ded3f (DEFAULT_TABLE table=default.Table_0931c459ae794d2091fcc0f51d031548 [id=e279c31bd63b40b9ade03a3b928bf153]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:14.665910 1235 leader_election.cc:290] T 25e85b0c5c9a4a3ca35a001359f21167 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:14.667989 1083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6448eea21ab2422c97da2494658ded3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:14.668845 816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25e85b0c5c9a4a3ca35a001359f21167" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:14.669474 816 raft_consensus.cc:2463] T 25e85b0c5c9a4a3ca35a001359f21167 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:14.670748 898 leader_election.cc:304] T 25e85b0c5c9a4a3ca35a001359f21167 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
I20250114 02:54:14.672115 1235 raft_consensus.cc:2798] T 25e85b0c5c9a4a3ca35a001359f21167 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:14.672513 1235 raft_consensus.cc:491] T 25e85b0c5c9a4a3ca35a001359f21167 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:14.672863 1235 raft_consensus.cc:3054] T 25e85b0c5c9a4a3ca35a001359f21167 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:14.679550 798 tablet_service.cc:1467] Processing CreateTablet for tablet 53a8bb487ad64157ad3baecf5d866eca (DEFAULT_TABLE table=default.Table_5d3b333f1d99463b8ad3b4318c8f29df [id=0f1c410e9c5c40ab8b8fe4e4bf68e8ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:14.679684 1505 consensus_queue.cc:1035] T 276942e5e54246c79f0d4f08297f58ea P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:14.680567 798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53a8bb487ad64157ad3baecf5d866eca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:14.698248 792 tablet_service.cc:1467] Processing CreateTablet for tablet 66bfa92db9874750b3c340e31a06c412 (DEFAULT_TABLE table=default.Table_b0cd93e5a498450e84fb0f1ba4be4505 [id=c70751b59bfc4c6fa60d0e8bb40d5f36]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:14.688475 1085 tablet_service.cc:1467] Processing CreateTablet for tablet 0f1ea1608213492c9a216757896ff2a4 (DEFAULT_TABLE table=default.Table_ae6b6fdfc1834cfb8089c860aedcc964 [id=7da003630a7a4e269d54fb2af8a3d4b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:14.698328 794 tablet_service.cc:1467] Processing CreateTablet for tablet 1381e26725174b9c877075cf5444f566 (DEFAULT_TABLE table=default.Table_569df7e8f022465c96cd2a326e905c72 [id=d2ea84ea59ad4b03ad16bb3052a441f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:14.699131 1205 tablet_bootstrap.cc:654] T c20aca761313498193e1d79c10705f8f P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:14.699345 792 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66bfa92db9874750b3c340e31a06c412. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:14.699343 1085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f1ea1608213492c9a216757896ff2a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:14.699926 794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1381e26725174b9c877075cf5444f566. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:14.709353 796 tablet_service.cc:1467] Processing CreateTablet for tablet 266da813b4514329be0ad97d07b7114f (DEFAULT_TABLE table=default.Table_ae220acb7e0345bdbb80b11398e4d153 [id=bcf9e834733d49a8bc235cd303a75a35]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:14.710390 796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 266da813b4514329be0ad97d07b7114f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:14.716449 1206 tablet_bootstrap.cc:654] T ed2d137579b8467da5eed4ea2f619a9a P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:14.723732 469 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_c6a60d505816422d91eccbe4813d7843" } new_table_name: "default.Table_0943b1ca347042c9a6043006bc566701" modify_external_catalogs: true
I20250114 02:54:14.734135 795 tablet_service.cc:1467] Processing CreateTablet for tablet bc46815b4803418c95ea9c039a4fae88 (DEFAULT_TABLE table=default.Table_6a4fe39298514187b65def5841d3a992 [id=5c5a7974ad2f4e58ba5c47a6eeaaac62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:14.735229 795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc46815b4803418c95ea9c039a4fae88. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:14.752533 1505 raft_consensus.cc:491] T 96b06fe4173343dabd726d253b9f9b5f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:14.753079 1505 raft_consensus.cc:513] T 96b06fe4173343dabd726d253b9f9b5f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:14.754848 1505 leader_election.cc:290] T 96b06fe4173343dabd726d253b9f9b5f P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:14.756137 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96b06fe4173343dabd726d253b9f9b5f" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
W20250114 02:54:14.757541 1040 leader_election.cc:343] T 96b06fe4173343dabd726d253b9f9b5f P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:14.777909 791 tablet_service.cc:1467] Processing CreateTablet for tablet 6448eea21ab2422c97da2494658ded3f (DEFAULT_TABLE table=default.Table_0931c459ae794d2091fcc0f51d031548 [id=e279c31bd63b40b9ade03a3b928bf153]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:14.779182 791 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6448eea21ab2422c97da2494658ded3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:14.755887 1408 raft_consensus.cc:513] T 83b5b36a26e74668998a19fa0ddf7188 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:14.791182 1408 leader_election.cc:290] T 83b5b36a26e74668998a19fa0ddf7188 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Requested vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:14.792585 816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83b5b36a26e74668998a19fa0ddf7188" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:14.793351 816 raft_consensus.cc:3054] T 83b5b36a26e74668998a19fa0ddf7188 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:14.806546 816 raft_consensus.cc:2463] T 83b5b36a26e74668998a19fa0ddf7188 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:14.807337 790 tablet_service.cc:1467] Processing CreateTablet for tablet 0f1ea1608213492c9a216757896ff2a4 (DEFAULT_TABLE table=default.Table_ae6b6fdfc1834cfb8089c860aedcc964 [id=7da003630a7a4e269d54fb2af8a3d4b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:14.808579 790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f1ea1608213492c9a216757896ff2a4. 1 dirs total, 0 dirs full, 0 dirs failed
W20250114 02:54:14.848193 1423 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
I20250114 02:54:14.848970 1386 raft_consensus.cc:695] T ed2d137579b8467da5eed4ea2f619a9a P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Becoming Leader. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Running, Role: LEADER
I20250114 02:54:14.852778 1386 consensus_queue.cc:237] T ed2d137579b8467da5eed4ea2f619a9a P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:14.856128 817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96b06fe4173343dabd726d253b9f9b5f" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
W20250114 02:54:14.857650 1423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 02:54:14.858100 1423 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250114 02:54:14.858510 1423 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250114 02:54:14.873418 1105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25e85b0c5c9a4a3ca35a001359f21167" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:14.874061 1105 raft_consensus.cc:2463] T 25e85b0c5c9a4a3ca35a001359f21167 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:14.874197 1496 raft_consensus.cc:491] T 179248cb5acf4740b268e0ec8b2852f9 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:14.874590 1496 raft_consensus.cc:513] T 179248cb5acf4740b268e0ec8b2852f9 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:14.876499 1496 leader_election.cc:290] T 179248cb5acf4740b268e0ec8b2852f9 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:14.877950 1105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "179248cb5acf4740b268e0ec8b2852f9" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:14.878582 1105 raft_consensus.cc:2463] T 179248cb5acf4740b268e0ec8b2852f9 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 0.
I20250114 02:54:14.879829 753 leader_election.cc:304] T 179248cb5acf4740b268e0ec8b2852f9 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:14.880934 1496 raft_consensus.cc:2798] T 179248cb5acf4740b268e0ec8b2852f9 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:14.881299 1496 raft_consensus.cc:491] T 179248cb5acf4740b268e0ec8b2852f9 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:14.881628 1496 raft_consensus.cc:3054] T 179248cb5acf4740b268e0ec8b2852f9 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:14.890949 1496 raft_consensus.cc:513] T 179248cb5acf4740b268e0ec8b2852f9 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:14.899495 1496 leader_election.cc:290] T 179248cb5acf4740b268e0ec8b2852f9 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Requested vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:14.900853 1105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "179248cb5acf4740b268e0ec8b2852f9" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:14.901487 1105 raft_consensus.cc:3054] T 179248cb5acf4740b268e0ec8b2852f9 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
W20250114 02:54:14.903388 1042 leader_election.cc:343] T 96b06fe4173343dabd726d253b9f9b5f P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:14.807809 898 leader_election.cc:304] T 83b5b36a26e74668998a19fa0ddf7188 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
W20250114 02:54:14.908684 898 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.65:39397, user_credentials={real_user=slave}} blocked reactor thread for 101023us
W20250114 02:54:14.910459 898 leader_election.cc:343] T 83b5b36a26e74668998a19fa0ddf7188 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Not found: Tablet not found: 83b5b36a26e74668998a19fa0ddf7188
I20250114 02:54:14.910619 959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "179248cb5acf4740b268e0ec8b2852f9" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:14.911811 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "179248cb5acf4740b268e0ec8b2852f9" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
I20250114 02:54:14.912377 1512 raft_consensus.cc:2798] T 83b5b36a26e74668998a19fa0ddf7188 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:14.912734 1105 raft_consensus.cc:2463] T 179248cb5acf4740b268e0ec8b2852f9 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 1.
W20250114 02:54:14.913542 751 leader_election.cc:343] T 179248cb5acf4740b268e0ec8b2852f9 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:14.914793 753 leader_election.cc:304] T 179248cb5acf4740b268e0ec8b2852f9 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:14.915321 936 tablet_service.cc:1467] Processing CreateTablet for tablet 6448eea21ab2422c97da2494658ded3f (DEFAULT_TABLE table=default.Table_0931c459ae794d2091fcc0f51d031548 [id=e279c31bd63b40b9ade03a3b928bf153]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:14.916468 936 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6448eea21ab2422c97da2494658ded3f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250114 02:54:14.916309 751 leader_election.cc:343] T 179248cb5acf4740b268e0ec8b2852f9 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:14.918990 1496 raft_consensus.cc:2798] T 179248cb5acf4740b268e0ec8b2852f9 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:14.920583 1496 raft_consensus.cc:695] T 179248cb5acf4740b268e0ec8b2852f9 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 LEADER]: Becoming Leader. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Running, Role: LEADER
I20250114 02:54:14.921823 1496 consensus_queue.cc:237] T 179248cb5acf4740b268e0ec8b2852f9 P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:14.941606 1510 consensus_queue.cc:1035] T b63dd90f60a04465b948171e74888e79 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:14.903795 1042 leader_election.cc:304] T 96b06fe4173343dabd726d253b9f9b5f P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90d6c2c58bb54e0eafb365625ac33871; no voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2
I20250114 02:54:14.945544 1205 tablet_bootstrap.cc:492] T c20aca761313498193e1d79c10705f8f P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:14.946202 1205 ts_tablet_manager.cc:1397] T c20aca761313498193e1d79c10705f8f P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.306s user 0.040s sys 0.038s
I20250114 02:54:14.887771 1235 raft_consensus.cc:513] T 25e85b0c5c9a4a3ca35a001359f21167 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
W20250114 02:54:14.956368 1042 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.65:39397, user_credentials={real_user=slave}} blocked reactor thread for 53007.7us
I20250114 02:54:14.958527 1235 leader_election.cc:290] T 25e85b0c5c9a4a3ca35a001359f21167 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Requested vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:14.961347 1507 raft_consensus.cc:2743] T 96b06fe4173343dabd726d253b9f9b5f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:14.963958 1507 raft_consensus.cc:491] T c20aca761313498193e1d79c10705f8f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:14.965446 1507 raft_consensus.cc:513] T c20aca761313498193e1d79c10705f8f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:14.969801 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c20aca761313498193e1d79c10705f8f" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
W20250114 02:54:14.971099 1040 leader_election.cc:343] T c20aca761313498193e1d79c10705f8f P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:14.972002 1512 raft_consensus.cc:695] T 83b5b36a26e74668998a19fa0ddf7188 P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Becoming Leader. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Running, Role: LEADER
I20250114 02:54:14.973080 1512 consensus_queue.cc:237] T 83b5b36a26e74668998a19fa0ddf7188 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:14.974634 1507 leader_election.cc:290] T c20aca761313498193e1d79c10705f8f P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:14.975488 1205 raft_consensus.cc:357] T c20aca761313498193e1d79c10705f8f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:14.976063 1205 raft_consensus.cc:383] T c20aca761313498193e1d79c10705f8f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:14.976385 1205 raft_consensus.cc:738] T c20aca761313498193e1d79c10705f8f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:14.976970 1205 consensus_queue.cc:260] T c20aca761313498193e1d79c10705f8f P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:14.977849 1206 tablet_bootstrap.cc:492] T ed2d137579b8467da5eed4ea2f619a9a P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:14.978230 1206 ts_tablet_manager.cc:1397] T ed2d137579b8467da5eed4ea2f619a9a P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.323s user 0.048s sys 0.027s
W20250114 02:54:14.979405 898 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.65:39397, user_credentials={real_user=slave}} blocked reactor thread for 50138.4us
I20250114 02:54:14.980412 1206 raft_consensus.cc:357] T ed2d137579b8467da5eed4ea2f619a9a P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:14.981056 1206 raft_consensus.cc:383] T ed2d137579b8467da5eed4ea2f619a9a P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:14.981303 1206 raft_consensus.cc:738] T ed2d137579b8467da5eed4ea2f619a9a P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:14.981818 1206 consensus_queue.cc:260] T ed2d137579b8467da5eed4ea2f619a9a P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:14.985157 1206 ts_tablet_manager.cc:1428] T ed2d137579b8467da5eed4ea2f619a9a P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20250114 02:54:14.987839 1496 consensus_queue.cc:1035] T 4bb9b429af92446bbe13894d5dfe4c47 P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:14.994072 1205 ts_tablet_manager.cc:1428] T c20aca761313498193e1d79c10705f8f P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.047s user 0.006s sys 0.000s
I20250114 02:54:14.998950 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25e85b0c5c9a4a3ca35a001359f21167" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:14.999680 818 raft_consensus.cc:3054] T 25e85b0c5c9a4a3ca35a001359f21167 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
W20250114 02:54:15.003490 1207 thread.cc:635] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.529s user 0.028s sys 0.065s
W20250114 02:54:15.003777 1207 thread.cc:601] log-alloc [worker] (thread pool) Time spent starting thread: real 0.529s user 0.028s sys 0.065s
I20250114 02:54:15.004725 1104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25e85b0c5c9a4a3ca35a001359f21167" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:15.005314 1104 raft_consensus.cc:3054] T 25e85b0c5c9a4a3ca35a001359f21167 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:15.006752 1423 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.107.84:40455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.107.126:45367,127.0.107.125:46837,127.0.107.124:39105
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.107.124:39105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.107.124
--webserver_port=36399
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type FASTDEBUG
built by None at 14 Jan 2025 02:43:23 UTC on a633016ff5c9
build id 4168
TSAN enabled
I20250114 02:54:15.008591 1423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 02:54:15.010355 1207 tablet_bootstrap.cc:492] T a3fdab330d574c0283ac58bc9e450b87 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:15.010797 1207 ts_tablet_manager.cc:1397] T a3fdab330d574c0283ac58bc9e450b87 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.545s user 0.035s sys 0.065s
I20250114 02:54:15.011886 1423 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250114 02:54:15.012483 1104 raft_consensus.cc:2463] T 25e85b0c5c9a4a3ca35a001359f21167 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:15.014604 815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c20aca761313498193e1d79c10705f8f" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:15.015215 815 raft_consensus.cc:2463] T c20aca761313498193e1d79c10705f8f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 0.
I20250114 02:54:15.022840 818 raft_consensus.cc:2463] T 25e85b0c5c9a4a3ca35a001359f21167 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:15.019068 1507 consensus_queue.cc:1035] T 276942e5e54246c79f0d4f08297f58ea P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20250114 02:54:15.025848 1207 raft_consensus.cc:357] T a3fdab330d574c0283ac58bc9e450b87 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:15.026495 1207 raft_consensus.cc:383] T a3fdab330d574c0283ac58bc9e450b87 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:15.026731 1207 raft_consensus.cc:738] T a3fdab330d574c0283ac58bc9e450b87 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:15.026849 1496 consensus_queue.cc:1035] T 4bb9b429af92446bbe13894d5dfe4c47 P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:54:15.021138 898 leader_election.cc:304] T 25e85b0c5c9a4a3ca35a001359f21167 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:15.021142 1510 consensus_queue.cc:1035] T b63dd90f60a04465b948171e74888e79 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:54:15.034631 1207 consensus_queue.cc:260] T a3fdab330d574c0283ac58bc9e450b87 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:15.035521 1510 raft_consensus.cc:2798] T 25e85b0c5c9a4a3ca35a001359f21167 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:15.036258 1510 raft_consensus.cc:695] T 25e85b0c5c9a4a3ca35a001359f21167 P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Becoming Leader. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Running, Role: LEADER
I20250114 02:54:15.037147 1510 consensus_queue.cc:237] T 25e85b0c5c9a4a3ca35a001359f21167 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
W20250114 02:54:15.040751 455 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.125:46837, user_credentials={real_user=slave}} blocked reactor thread for 368976us
I20250114 02:54:15.053144 940 tablet_service.cc:1467] Processing CreateTablet for tablet 0f1ea1608213492c9a216757896ff2a4 (DEFAULT_TABLE table=default.Table_ae6b6fdfc1834cfb8089c860aedcc964 [id=7da003630a7a4e269d54fb2af8a3d4b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:15.054527 940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f1ea1608213492c9a216757896ff2a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:15.070196 1205 tablet_bootstrap.cc:492] T 9589afd0094545c1801b7a0b5eb8698f P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:15.081789 1505 raft_consensus.cc:491] T a3fdab330d574c0283ac58bc9e450b87 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:15.081944 1207 ts_tablet_manager.cc:1428] T a3fdab330d574c0283ac58bc9e450b87 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.071s user 0.008s sys 0.000s
I20250114 02:54:15.082249 1505 raft_consensus.cc:513] T a3fdab330d574c0283ac58bc9e450b87 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:15.082736 1207 tablet_bootstrap.cc:492] T 290ad07ab8be4791a39194909e000843 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:15.087045 959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3fdab330d574c0283ac58bc9e450b87" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:15.087836 1207 tablet_bootstrap.cc:654] T 290ad07ab8be4791a39194909e000843 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
W20250114 02:54:15.088515 1040 leader_election.cc:343] T a3fdab330d574c0283ac58bc9e450b87 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:15.084095 1505 leader_election.cc:290] T a3fdab330d574c0283ac58bc9e450b87 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:15.108520 1205 tablet_bootstrap.cc:654] T 9589afd0094545c1801b7a0b5eb8698f P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:15.130857 1536 raft_consensus.cc:491] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b303b7df086340a0b5ad6373940ae750)
I20250114 02:54:15.131311 1536 raft_consensus.cc:513] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b303b7df086340a0b5ad6373940ae750" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 45367 } } peers { permanent_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } } peers { permanent_uuid: "87b3db09ac87436e9a60d6de5fedaff5" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } }
I20250114 02:54:15.135293 481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" candidate_term: 2 candidate_status { last_received { term: 1 index: 188 } } ignore_live_leader: false dest_uuid: "b303b7df086340a0b5ad6373940ae750" is_pre_election: true
I20250114 02:54:15.156944 1536 leader_election.cc:290] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b303b7df086340a0b5ad6373940ae750 (127.0.107.126:45367), 87b3db09ac87436e9a60d6de5fedaff5 (127.0.107.124:39105)
W20250114 02:54:15.165727 1042 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.65:39397, user_credentials={real_user=slave}} blocked reactor thread for 148053us
I20250114 02:54:15.167312 1042 leader_election.cc:304] T c20aca761313498193e1d79c10705f8f P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters: 525198593c8e47afaee54e74267b87e3
I20250114 02:54:15.168048 1507 raft_consensus.cc:2798] T c20aca761313498193e1d79c10705f8f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:15.168413 1507 raft_consensus.cc:491] T c20aca761313498193e1d79c10705f8f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:15.168728 1507 raft_consensus.cc:3054] T c20aca761313498193e1d79c10705f8f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:15.192791 1507 raft_consensus.cc:513] T c20aca761313498193e1d79c10705f8f P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:15.194664 1507 leader_election.cc:290] T c20aca761313498193e1d79c10705f8f P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Requested vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:15.195394 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c20aca761313498193e1d79c10705f8f" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
I20250114 02:54:15.195928 962 raft_consensus.cc:3054] T c20aca761313498193e1d79c10705f8f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:15.198185 1207 tablet_bootstrap.cc:492] T 290ad07ab8be4791a39194909e000843 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:15.198824 1207 ts_tablet_manager.cc:1397] T 290ad07ab8be4791a39194909e000843 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.116s user 0.012s sys 0.017s
I20250114 02:54:15.205150 962 raft_consensus.cc:2463] T c20aca761313498193e1d79c10705f8f P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1.
I20250114 02:54:15.205698 1207 raft_consensus.cc:357] T 290ad07ab8be4791a39194909e000843 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:15.206588 1040 leader_election.cc:304] T c20aca761313498193e1d79c10705f8f P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:15.208115 1505 raft_consensus.cc:2798] T c20aca761313498193e1d79c10705f8f P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:15.209285 1505 raft_consensus.cc:695] T c20aca761313498193e1d79c10705f8f P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Becoming Leader. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Running, Role: LEADER
I20250114 02:54:15.206606 1207 raft_consensus.cc:383] T 290ad07ab8be4791a39194909e000843 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:15.209910 1207 raft_consensus.cc:738] T 290ad07ab8be4791a39194909e000843 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:15.211381 1207 consensus_queue.cc:260] T 290ad07ab8be4791a39194909e000843 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:15.212574 1505 consensus_queue.cc:237] T c20aca761313498193e1d79c10705f8f P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
W20250114 02:54:15.223739 530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.107.124:39105: connect: Connection refused (error 111) [suppressed 6 similar messages]
I20250114 02:54:15.225340 1207 ts_tablet_manager.cc:1428] T 290ad07ab8be4791a39194909e000843 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.026s user 0.007s sys 0.000s
I20250114 02:54:15.226465 1207 tablet_bootstrap.cc:492] T 15704785f56e43a3af69f181c75877ce P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:15.228725 1206 tablet_bootstrap.cc:492] T 9bfd8fd6d2104fb4a6484cb8c98981bd P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:15.250310 1207 tablet_bootstrap.cc:654] T 15704785f56e43a3af69f181c75877ce P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
W20250114 02:54:15.266373 530 leader_election.cc:336] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 87b3db09ac87436e9a60d6de5fedaff5 (127.0.107.124:39105): Network error: Client connection negotiation failed: client connection to 127.0.107.124:39105: connect: Connection refused (error 111)
I20250114 02:54:15.266940 1507 raft_consensus.cc:491] T 290ad07ab8be4791a39194909e000843 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:15.267486 1507 raft_consensus.cc:513] T 290ad07ab8be4791a39194909e000843 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:15.268978 1206 tablet_bootstrap.cc:654] T 9bfd8fd6d2104fb4a6484cb8c98981bd P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:15.270563 1507 leader_election.cc:290] T 290ad07ab8be4791a39194909e000843 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:15.272207 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "290ad07ab8be4791a39194909e000843" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
W20250114 02:54:15.273514 1040 leader_election.cc:343] T 290ad07ab8be4791a39194909e000843 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:15.273784 1207 tablet_bootstrap.cc:492] T 15704785f56e43a3af69f181c75877ce P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:15.274253 1207 ts_tablet_manager.cc:1397] T 15704785f56e43a3af69f181c75877ce P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.048s user 0.011s sys 0.000s
I20250114 02:54:15.275936 530 leader_election.cc:304] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 687d4ff6d3cb42cf90a965070b0d8a70; no voters: 87b3db09ac87436e9a60d6de5fedaff5, b303b7df086340a0b5ad6373940ae750
I20250114 02:54:15.276798 1536 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:54:15.277235 1207 raft_consensus.cc:357] T 15704785f56e43a3af69f181c75877ce P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:15.278052 1207 raft_consensus.cc:383] T 15704785f56e43a3af69f181c75877ce P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:15.278349 1207 raft_consensus.cc:738] T 15704785f56e43a3af69f181c75877ce P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
W20250114 02:54:15.286454 1531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 02:54:15.289584 1205 tablet_bootstrap.cc:492] T 9589afd0094545c1801b7a0b5eb8698f P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:15.289958 1205 ts_tablet_manager.cc:1397] T 9589afd0094545c1801b7a0b5eb8698f P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.220s user 0.047s sys 0.104s
I20250114 02:54:15.291947 1205 raft_consensus.cc:357] T 9589afd0094545c1801b7a0b5eb8698f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:15.292568 1205 raft_consensus.cc:383] T 9589afd0094545c1801b7a0b5eb8698f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:15.292790 1205 raft_consensus.cc:738] T 9589afd0094545c1801b7a0b5eb8698f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:15.292624 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3fdab330d574c0283ac58bc9e450b87" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:15.293298 1205 consensus_queue.cc:260] T 9589afd0094545c1801b7a0b5eb8698f P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:15.295887 816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c20aca761313498193e1d79c10705f8f" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:15.296481 816 raft_consensus.cc:3054] T c20aca761313498193e1d79c10705f8f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:15.296537 1207 consensus_queue.cc:260] T 15704785f56e43a3af69f181c75877ce P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:15.299674 1207 ts_tablet_manager.cc:1428] T 15704785f56e43a3af69f181c75877ce P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.025s user 0.006s sys 0.001s
I20250114 02:54:15.300503 1207 tablet_bootstrap.cc:492] T e3af8f1299f24b08b2d25811942aaecc P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:15.305341 1205 ts_tablet_manager.cc:1428] T 9589afd0094545c1801b7a0b5eb8698f P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.015s user 0.005s sys 0.001s
I20250114 02:54:15.310613 1207 tablet_bootstrap.cc:654] T e3af8f1299f24b08b2d25811942aaecc P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:15.313797 1205 tablet_bootstrap.cc:492] T 9bfd8fd6d2104fb4a6484cb8c98981bd P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:15.313907 815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "290ad07ab8be4791a39194909e000843" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:15.321008 816 raft_consensus.cc:2463] T c20aca761313498193e1d79c10705f8f P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1.
I20250114 02:54:15.324501 1205 tablet_bootstrap.cc:654] T 9bfd8fd6d2104fb4a6484cb8c98981bd P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:15.326495 1207 tablet_bootstrap.cc:492] T e3af8f1299f24b08b2d25811942aaecc P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:15.326934 1207 ts_tablet_manager.cc:1397] T e3af8f1299f24b08b2d25811942aaecc P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.027s user 0.008s sys 0.000s
I20250114 02:54:15.329609 1207 raft_consensus.cc:357] T e3af8f1299f24b08b2d25811942aaecc P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:15.330461 1207 raft_consensus.cc:383] T e3af8f1299f24b08b2d25811942aaecc P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:15.330766 1207 raft_consensus.cc:738] T e3af8f1299f24b08b2d25811942aaecc P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:15.331434 962 raft_consensus.cc:3054] T 9589afd0094545c1801b7a0b5eb8698f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:15.336795 1205 tablet_bootstrap.cc:492] T 9bfd8fd6d2104fb4a6484cb8c98981bd P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:15.337157 1205 ts_tablet_manager.cc:1397] T 9bfd8fd6d2104fb4a6484cb8c98981bd P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.004s
I20250114 02:54:15.339273 1205 raft_consensus.cc:357] T 9bfd8fd6d2104fb4a6484cb8c98981bd P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:15.339893 1205 raft_consensus.cc:383] T 9bfd8fd6d2104fb4a6484cb8c98981bd P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:15.340123 1205 raft_consensus.cc:738] T 9bfd8fd6d2104fb4a6484cb8c98981bd P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:15.340631 1205 consensus_queue.cc:260] T 9bfd8fd6d2104fb4a6484cb8c98981bd P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
W20250114 02:54:15.342053 1042 leader_election.cc:343] T a3fdab330d574c0283ac58bc9e450b87 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:15.342562 1042 leader_election.cc:304] T a3fdab330d574c0283ac58bc9e450b87 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90d6c2c58bb54e0eafb365625ac33871; no voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2
I20250114 02:54:15.343717 1507 raft_consensus.cc:2743] T a3fdab330d574c0283ac58bc9e450b87 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250114 02:54:15.344820 1042 leader_election.cc:343] T 290ad07ab8be4791a39194909e000843 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:15.331472 1207 consensus_queue.cc:260] T e3af8f1299f24b08b2d25811942aaecc P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:15.345213 1042 leader_election.cc:304] T 290ad07ab8be4791a39194909e000843 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90d6c2c58bb54e0eafb365625ac33871; no voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2
I20250114 02:54:15.350034 1507 raft_consensus.cc:2743] T 290ad07ab8be4791a39194909e000843 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:15.351663 1207 ts_tablet_manager.cc:1428] T e3af8f1299f24b08b2d25811942aaecc P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.024s user 0.007s sys 0.000s
I20250114 02:54:15.352705 1207 tablet_bootstrap.cc:492] T de4ca83a69704a368a25954cdc1b9392 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:15.353155 1205 ts_tablet_manager.cc:1428] T 9bfd8fd6d2104fb4a6484cb8c98981bd P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20250114 02:54:15.352932 1548 consensus_queue.cc:1035] T 9589afd0094545c1801b7a0b5eb8698f P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:15.354863 1205 tablet_bootstrap.cc:492] T 179248cb5acf4740b268e0ec8b2852f9 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:15.369391 1205 tablet_bootstrap.cc:654] T 179248cb5acf4740b268e0ec8b2852f9 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:15.385738 1207 tablet_bootstrap.cc:654] T de4ca83a69704a368a25954cdc1b9392 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:15.390666 1205 tablet_bootstrap.cc:492] T 179248cb5acf4740b268e0ec8b2852f9 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:15.391086 1205 ts_tablet_manager.cc:1397] T 179248cb5acf4740b268e0ec8b2852f9 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.036s user 0.007s sys 0.003s
I20250114 02:54:15.396152 1205 raft_consensus.cc:357] T 179248cb5acf4740b268e0ec8b2852f9 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:15.396852 1205 raft_consensus.cc:383] T 179248cb5acf4740b268e0ec8b2852f9 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:15.397084 1205 raft_consensus.cc:738] T 179248cb5acf4740b268e0ec8b2852f9 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:15.397610 1205 consensus_queue.cc:260] T 179248cb5acf4740b268e0ec8b2852f9 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:15.409670 1205 ts_tablet_manager.cc:1428] T 179248cb5acf4740b268e0ec8b2852f9 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20250114 02:54:15.410019 1207 tablet_bootstrap.cc:492] T de4ca83a69704a368a25954cdc1b9392 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:15.410526 1207 ts_tablet_manager.cc:1397] T de4ca83a69704a368a25954cdc1b9392 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.058s user 0.007s sys 0.004s
I20250114 02:54:15.413071 1207 raft_consensus.cc:357] T de4ca83a69704a368a25954cdc1b9392 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:15.413838 1207 raft_consensus.cc:383] T de4ca83a69704a368a25954cdc1b9392 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:15.412446 1205 tablet_bootstrap.cc:492] T 96b06fe4173343dabd726d253b9f9b5f P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:15.414119 1207 raft_consensus.cc:738] T de4ca83a69704a368a25954cdc1b9392 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:15.416322 1207 consensus_queue.cc:260] T de4ca83a69704a368a25954cdc1b9392 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:15.418062 1207 ts_tablet_manager.cc:1428] T de4ca83a69704a368a25954cdc1b9392 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250114 02:54:15.418834 1207 tablet_bootstrap.cc:492] T 961b6a25a13a4fa2a732a6383dcd8c84 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:15.420471 1205 tablet_bootstrap.cc:654] T 96b06fe4173343dabd726d253b9f9b5f P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:15.436085 1207 tablet_bootstrap.cc:654] T 961b6a25a13a4fa2a732a6383dcd8c84 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:15.437381 465 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33540 (request call id 152) took 1010 ms. Trace:
I20250114 02:54:15.438868 1205 tablet_bootstrap.cc:492] T 96b06fe4173343dabd726d253b9f9b5f P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:15.437673 465 rpcz_store.cc:276] 0114 02:54:14.426572 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:14.427308 (+ 736us) service_pool.cc:225] Handling call
0114 02:54:14.436345 (+ 9037us) spinlock_profiling.cc:243] Waited 8.04 ms on lock 0x7b54000316d4. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616cad50 00007f646174df2a 00007f64633f2dec 00007f64634ec306 00007f6461250044 00007f646124ffd2 00007f646124ff01 00007f646124fe7c 00007f646124f132 00007f645f512dfc 00007f645f512236
0114 02:54:14.438375 (+ 2030us) catalog_manager.cc:2076] Acquired catalog manager lock
0114 02:54:14.439698 (+ 1323us) catalog_manager.cc:2122] Created new table and tablet info
0114 02:54:15.435906 (+996208us) catalog_manager.cc:2182] Wrote table and tablets to system table
0114 02:54:15.436900 (+ 994us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0114 02:54:15.437221 (+ 321us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:15.075200 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:15.075715 (+ 515us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:15.075750 (+ 35us) write_op.cc:435] Acquired schema lock
0114 02:54:15.075769 (+ 19us) tablet.cc:659] Decoding operations
0114 02:54:15.076084 (+ 315us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:15.076130 (+ 46us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:15.076158 (+ 28us) tablet.cc:682] Acquiring locks for 2 operations
0114 02:54:15.076504 (+ 346us) tablet.cc:698] Row locks acquired
0114 02:54:15.076536 (+ 32us) write_op.cc:260] PREPARE: finished
0114 02:54:15.076719 (+ 183us) write_op.cc:270] Start()
0114 02:54:15.076846 (+ 127us) write_op.cc:276] Timestamp: P: 1736823255076593 usec, L: 0
0114 02:54:15.076873 (+ 27us) op_driver.cc:348] REPLICATION: starting
0114 02:54:15.078146 (+ 1273us) log.cc:844] Serialized 433 byte log entry
0114 02:54:15.432473 (+354327us) op_driver.cc:464] REPLICATION: finished
0114 02:54:15.433510 (+ 1037us) write_op.cc:301] APPLY: starting
0114 02:54:15.433565 (+ 55us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:15.433665 (+ 100us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:15.433777 (+ 112us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:15.434247 (+ 470us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:15.434422 (+ 175us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 02:54:15.434475 (+ 53us) write_op.cc:312] APPLY: finished
0114 02:54:15.434691 (+ 216us) log.cc:844] Serialized 29 byte log entry
0114 02:54:15.435538 (+ 847us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:15.435607 (+ 69us) write_op.cc:454] Released schema lock
0114 02:54:15.435651 (+ 44us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":18480896,"child_traces":[["op",{"apply.queue_time_us":863,"num_ops":2,"prepare.queue_time_us":634281,"prepare.run_cpu_time_us":3229,"prepare.run_wall_time_us":3228,"replication_time_us":355512,"threads_started":1}]]}
I20250114 02:54:15.439258 1205 ts_tablet_manager.cc:1397] T 96b06fe4173343dabd726d253b9f9b5f P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.027s user 0.009s sys 0.002s
I20250114 02:54:15.448123 1205 raft_consensus.cc:357] T 96b06fe4173343dabd726d253b9f9b5f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:15.448798 1205 raft_consensus.cc:383] T 96b06fe4173343dabd726d253b9f9b5f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:15.451197 1207 tablet_bootstrap.cc:492] T 961b6a25a13a4fa2a732a6383dcd8c84 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:15.450510 1205 raft_consensus.cc:738] T 96b06fe4173343dabd726d253b9f9b5f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:15.451584 1207 ts_tablet_manager.cc:1397] T 961b6a25a13a4fa2a732a6383dcd8c84 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.033s user 0.009s sys 0.002s
I20250114 02:54:15.452306 1205 consensus_queue.cc:260] T 96b06fe4173343dabd726d253b9f9b5f P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:15.453996 1207 raft_consensus.cc:357] T 961b6a25a13a4fa2a732a6383dcd8c84 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:15.454643 1207 raft_consensus.cc:383] T 961b6a25a13a4fa2a732a6383dcd8c84 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:15.454886 1207 raft_consensus.cc:738] T 961b6a25a13a4fa2a732a6383dcd8c84 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:15.455411 1207 consensus_queue.cc:260] T 961b6a25a13a4fa2a732a6383dcd8c84 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:15.457403 1207 ts_tablet_manager.cc:1428] T 961b6a25a13a4fa2a732a6383dcd8c84 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20250114 02:54:15.457491 1205 ts_tablet_manager.cc:1428] T 96b06fe4173343dabd726d253b9f9b5f P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.013s user 0.004s sys 0.000s
I20250114 02:54:15.465420 470 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33540 (request call id 150) took 1171 ms. Trace:
I20250114 02:54:15.465754 470 rpcz_store.cc:276] 0114 02:54:14.294136 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:14.297667 (+ 3531us) service_pool.cc:225] Handling call
0114 02:54:14.300068 (+ 2401us) catalog_manager.cc:2076] Acquired catalog manager lock
0114 02:54:14.358116 (+ 58048us) catalog_manager.cc:2122] Created new table and tablet info
0114 02:54:15.464512 (+1106396us) catalog_manager.cc:2182] Wrote table and tablets to system table
0114 02:54:15.464954 (+ 442us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0114 02:54:15.465263 (+ 309us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:14.726858 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:14.727168 (+ 310us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:14.727189 (+ 21us) write_op.cc:435] Acquired schema lock
0114 02:54:14.727201 (+ 12us) tablet.cc:659] Decoding operations
0114 02:54:14.727441 (+ 240us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:14.727476 (+ 35us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:14.727497 (+ 21us) tablet.cc:682] Acquiring locks for 2 operations
0114 02:54:14.727664 (+ 167us) tablet.cc:698] Row locks acquired
0114 02:54:14.727676 (+ 12us) write_op.cc:260] PREPARE: finished
0114 02:54:14.727773 (+ 97us) write_op.cc:270] Start()
0114 02:54:14.727874 (+ 101us) write_op.cc:276] Timestamp: P: 1736823254727701 usec, L: 0
0114 02:54:14.727887 (+ 13us) op_driver.cc:348] REPLICATION: starting
0114 02:54:14.728442 (+ 555us) log.cc:844] Serialized 433 byte log entry
0114 02:54:15.183025 (+454583us) op_driver.cc:464] REPLICATION: finished
0114 02:54:15.410027 (+227002us) write_op.cc:301] APPLY: starting
0114 02:54:15.410128 (+ 101us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:15.410252 (+ 124us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:15.459560 (+ 49308us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:15.460139 (+ 579us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:15.460332 (+ 193us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 02:54:15.460371 (+ 39us) write_op.cc:312] APPLY: finished
0114 02:54:15.460560 (+ 189us) log.cc:844] Serialized 29 byte log entry
0114 02:54:15.460734 (+ 174us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:15.460822 (+ 88us) write_op.cc:454] Released schema lock
0114 02:54:15.460856 (+ 34us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:15.461144 (+ 288us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:15.461177 (+ 33us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":226765,"apply.run_cpu_time_us":1925,"apply.run_wall_time_us":51079,"num_ops":2,"prepare.queue_time_us":367563,"prepare.run_cpu_time_us":2062,"prepare.run_wall_time_us":348060,"replication_time_us":454997,"thread_start_us":248974,"threads_started":1}]]}
I20250114 02:54:15.469395 470 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_1c87461bd6f64248aecf8fd2d9d3700b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:15.473927 753 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.67:45749, user_credentials={real_user=slave}} blocked reactor thread for 221864us
W20250114 02:54:15.475577 470 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1c87461bd6f64248aecf8fd2d9d3700b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:15.480674 1205 tablet_bootstrap.cc:492] T e3af8f1299f24b08b2d25811942aaecc P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:15.481345 1206 tablet_bootstrap.cc:492] T 9bfd8fd6d2104fb4a6484cb8c98981bd P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:15.481866 1206 ts_tablet_manager.cc:1397] T 9bfd8fd6d2104fb4a6484cb8c98981bd P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.256s user 0.010s sys 0.004s
I20250114 02:54:15.485947 1205 tablet_bootstrap.cc:654] T e3af8f1299f24b08b2d25811942aaecc P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:15.508945 1206 raft_consensus.cc:357] T 9bfd8fd6d2104fb4a6484cb8c98981bd P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:15.509792 1206 raft_consensus.cc:383] T 9bfd8fd6d2104fb4a6484cb8c98981bd P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:15.510116 1206 raft_consensus.cc:738] T 9bfd8fd6d2104fb4a6484cb8c98981bd P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:15.513667 1205 tablet_bootstrap.cc:492] T e3af8f1299f24b08b2d25811942aaecc P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:15.514031 1205 ts_tablet_manager.cc:1397] T e3af8f1299f24b08b2d25811942aaecc P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.034s user 0.006s sys 0.008s
I20250114 02:54:15.514381 1206 consensus_queue.cc:260] T 9bfd8fd6d2104fb4a6484cb8c98981bd P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:15.516045 1205 raft_consensus.cc:357] T e3af8f1299f24b08b2d25811942aaecc P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:15.516839 1205 raft_consensus.cc:383] T e3af8f1299f24b08b2d25811942aaecc P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:15.517097 1205 raft_consensus.cc:738] T e3af8f1299f24b08b2d25811942aaecc P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:15.518332 1205 consensus_queue.cc:260] T e3af8f1299f24b08b2d25811942aaecc P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
W20250114 02:54:15.519696 456 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 179248cb5acf4740b268e0ec8b2852f9 (table default.Table_bd405598fe054b718a8d6fb5dd875f0e [id=5745501ecd7b4cecaca3d35502a6cc40], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 179248cb5acf4740b268e0ec8b2852f9 consensus state has no leader
I20250114 02:54:15.523818 1206 ts_tablet_manager.cc:1428] T 9bfd8fd6d2104fb4a6484cb8c98981bd P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.042s user 0.005s sys 0.001s
I20250114 02:54:15.524866 1206 tablet_bootstrap.cc:492] T 96b06fe4173343dabd726d253b9f9b5f P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:15.529340 1205 ts_tablet_manager.cc:1428] T e3af8f1299f24b08b2d25811942aaecc P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.015s user 0.006s sys 0.001s
I20250114 02:54:15.532294 1205 tablet_bootstrap.cc:492] T a3fdab330d574c0283ac58bc9e450b87 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:15.538056 1205 tablet_bootstrap.cc:654] T a3fdab330d574c0283ac58bc9e450b87 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:15.546891 1206 tablet_bootstrap.cc:654] T 96b06fe4173343dabd726d253b9f9b5f P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:15.549326 1205 tablet_bootstrap.cc:492] T a3fdab330d574c0283ac58bc9e450b87 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:15.549738 1205 ts_tablet_manager.cc:1397] T a3fdab330d574c0283ac58bc9e450b87 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.000s
I20250114 02:54:15.552372 1205 raft_consensus.cc:357] T a3fdab330d574c0283ac58bc9e450b87 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:15.553673 1205 raft_consensus.cc:383] T a3fdab330d574c0283ac58bc9e450b87 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:15.553929 1205 raft_consensus.cc:738] T a3fdab330d574c0283ac58bc9e450b87 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:15.555598 1104 raft_consensus.cc:3054] T 268a12df225e4ba59d4a84c2869d4661 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:15.558604 1205 consensus_queue.cc:260] T a3fdab330d574c0283ac58bc9e450b87 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:15.566936 1548 consensus_queue.cc:1035] T 268a12df225e4ba59d4a84c2869d4661 P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:54:15.571043 1548 consensus_queue.cc:1035] T 9589afd0094545c1801b7a0b5eb8698f P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:15.589030 1205 ts_tablet_manager.cc:1428] T a3fdab330d574c0283ac58bc9e450b87 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.039s user 0.005s sys 0.000s
I20250114 02:54:15.592731 1386 raft_consensus.cc:491] T e3af8f1299f24b08b2d25811942aaecc P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:15.593477 1386 raft_consensus.cc:513] T e3af8f1299f24b08b2d25811942aaecc P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:15.597702 1207 tablet_bootstrap.cc:492] T 7ecf782bb3d54528aa996abca1ea3a15 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:15.601018 1205 tablet_bootstrap.cc:492] T 290ad07ab8be4791a39194909e000843 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:15.602922 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3af8f1299f24b08b2d25811942aaecc" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
W20250114 02:54:15.605744 1042 leader_election.cc:343] T e3af8f1299f24b08b2d25811942aaecc P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:15.606191 1386 leader_election.cc:290] T e3af8f1299f24b08b2d25811942aaecc P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:15.609197 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3af8f1299f24b08b2d25811942aaecc" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:15.609731 962 raft_consensus.cc:2463] T e3af8f1299f24b08b2d25811942aaecc P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 0.
I20250114 02:54:15.611192 1040 leader_election.cc:304] T e3af8f1299f24b08b2d25811942aaecc P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters: 819f9e3412e54d44abb11d8b3ca877c2
I20250114 02:54:15.617507 1205 tablet_bootstrap.cc:654] T 290ad07ab8be4791a39194909e000843 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:15.625042 1386 raft_consensus.cc:2798] T e3af8f1299f24b08b2d25811942aaecc P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:15.626171 1386 raft_consensus.cc:491] T e3af8f1299f24b08b2d25811942aaecc P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:15.626865 1386 raft_consensus.cc:3054] T e3af8f1299f24b08b2d25811942aaecc P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:15.633282 1207 tablet_bootstrap.cc:654] T 7ecf782bb3d54528aa996abca1ea3a15 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:15.636951 1205 tablet_bootstrap.cc:492] T 290ad07ab8be4791a39194909e000843 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:15.637390 1205 ts_tablet_manager.cc:1397] T 290ad07ab8be4791a39194909e000843 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.037s user 0.007s sys 0.004s
I20250114 02:54:15.639295 1206 tablet_bootstrap.cc:492] T 96b06fe4173343dabd726d253b9f9b5f P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:15.639806 1206 ts_tablet_manager.cc:1397] T 96b06fe4173343dabd726d253b9f9b5f P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.115s user 0.027s sys 0.012s
I20250114 02:54:15.642503 1206 raft_consensus.cc:357] T 96b06fe4173343dabd726d253b9f9b5f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:15.643137 1206 raft_consensus.cc:383] T 96b06fe4173343dabd726d253b9f9b5f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:15.643379 1206 raft_consensus.cc:738] T 96b06fe4173343dabd726d253b9f9b5f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:15.645499 1505 raft_consensus.cc:491] T de4ca83a69704a368a25954cdc1b9392 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:15.645920 1505 raft_consensus.cc:513] T de4ca83a69704a368a25954cdc1b9392 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:15.647869 1505 leader_election.cc:290] T de4ca83a69704a368a25954cdc1b9392 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:15.649350 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de4ca83a69704a368a25954cdc1b9392" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:15.649943 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de4ca83a69704a368a25954cdc1b9392" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
W20250114 02:54:15.650836 1040 leader_election.cc:343] T de4ca83a69704a368a25954cdc1b9392 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
W20250114 02:54:15.651340 1042 leader_election.cc:343] T de4ca83a69704a368a25954cdc1b9392 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:15.651780 1042 leader_election.cc:304] T de4ca83a69704a368a25954cdc1b9392 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90d6c2c58bb54e0eafb365625ac33871; no voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2
I20250114 02:54:15.652901 467 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:33540 (request call id 154) took 1135 ms. Trace:
I20250114 02:54:15.652909 1508 raft_consensus.cc:2743] T de4ca83a69704a368a25954cdc1b9392 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:15.665339 471 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33540 (request call id 153) took 1238 ms. Trace:
I20250114 02:54:15.666026 1206 consensus_queue.cc:260] T 96b06fe4173343dabd726d253b9f9b5f P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:15.665707 471 rpcz_store.cc:276] 0114 02:54:14.427206 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:14.473253 (+ 46047us) service_pool.cc:225] Handling call
0114 02:54:14.499126 (+ 25873us) spinlock_profiling.cc:243] Waited 25.1 ms on lock 0x7b54000316d4. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616cad50 00007f646174df2a 00007f64633f2dec 00007f64634ec306 00007f6461250044 00007f646124ffd2 00007f646124ff01 00007f646124fe7c 00007f646124f132 00007f645f512dfc 00007f645f512236
0114 02:54:14.501269 (+ 2143us) catalog_manager.cc:2076] Acquired catalog manager lock
0114 02:54:14.502642 (+ 1373us) catalog_manager.cc:2122] Created new table and tablet info
0114 02:54:15.664404 (+1161762us) catalog_manager.cc:2182] Wrote table and tablets to system table
0114 02:54:15.664809 (+ 405us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0114 02:54:15.665147 (+ 338us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:15.461824 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:15.462255 (+ 431us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:15.462275 (+ 20us) write_op.cc:435] Acquired schema lock
0114 02:54:15.462292 (+ 17us) tablet.cc:659] Decoding operations
0114 02:54:15.462554 (+ 262us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:15.462583 (+ 29us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:15.462603 (+ 20us) tablet.cc:682] Acquiring locks for 2 operations
0114 02:54:15.462795 (+ 192us) tablet.cc:698] Row locks acquired
0114 02:54:15.462807 (+ 12us) write_op.cc:260] PREPARE: finished
0114 02:54:15.462934 (+ 127us) write_op.cc:270] Start()
0114 02:54:15.463017 (+ 83us) write_op.cc:276] Timestamp: P: 1736823255462787 usec, L: 0
0114 02:54:15.463041 (+ 24us) op_driver.cc:348] REPLICATION: starting
0114 02:54:15.463511 (+ 470us) log.cc:844] Serialized 433 byte log entry
0114 02:54:15.646125 (+182614us) op_driver.cc:464] REPLICATION: finished
0114 02:54:15.649430 (+ 3305us) write_op.cc:301] APPLY: starting
0114 02:54:15.649473 (+ 43us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:15.649558 (+ 85us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:15.661760 (+ 12202us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:15.662183 (+ 423us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:15.662347 (+ 164us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 02:54:15.662399 (+ 52us) write_op.cc:312] APPLY: finished
0114 02:54:15.662574 (+ 175us) log.cc:844] Serialized 29 byte log entry
0114 02:54:15.663108 (+ 534us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:15.663189 (+ 81us) write_op.cc:454] Released schema lock
0114 02:54:15.663242 (+ 53us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":57803008,"child_traces":[["op",{"apply.queue_time_us":3200,"num_ops":2,"prepare.queue_time_us":833222,"prepare.run_cpu_time_us":1873,"prepare.run_wall_time_us":1872,"replication_time_us":183010,"threads_started":1}]]}
I20250114 02:54:15.663370 463 catalog_manager.cc:5526] T b63dd90f60a04465b948171e74888e79 P 525198593c8e47afaee54e74267b87e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 525198593c8e47afaee54e74267b87e3 (127.0.107.66). New cstate: current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } }
I20250114 02:54:15.674350 463 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet b63dd90f60a04465b948171e74888e79 (table default.Table_1097f09fb10e441fb639f8d7aa12d487 [id=d8c9691818d042d7b49475fa009c4ca1]). Expected version 3 got 0
I20250114 02:54:15.653211 467 rpcz_store.cc:276] 0114 02:54:14.517616 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:14.520555 (+ 2939us) service_pool.cc:225] Handling call
0114 02:54:14.521413 (+ 858us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0114 02:54:14.522051 (+ 638us) catalog_manager.cc:3627] Apply alter schema
0114 02:54:14.522552 (+ 501us) catalog_manager.cc:3659] Acquired catalog manager lock
0114 02:54:14.522743 (+ 191us) catalog_manager.cc:3817] Updating metadata on disk
0114 02:54:15.651387 (+1128644us) catalog_manager.cc:3848] Committing alterations to in-memory state
0114 02:54:15.652680 (+ 1293us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:15.183827 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:15.184401 (+ 574us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:15.184431 (+ 30us) write_op.cc:435] Acquired schema lock
0114 02:54:15.184450 (+ 19us) tablet.cc:659] Decoding operations
0114 02:54:15.184796 (+ 346us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:15.184842 (+ 46us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:15.184875 (+ 33us) tablet.cc:682] Acquiring locks for 1 operations
0114 02:54:15.185049 (+ 174us) tablet.cc:698] Row locks acquired
0114 02:54:15.185069 (+ 20us) write_op.cc:260] PREPARE: finished
0114 02:54:15.185208 (+ 139us) write_op.cc:270] Start()
0114 02:54:15.185306 (+ 98us) write_op.cc:276] Timestamp: P: 1736823255185071 usec, L: 0
0114 02:54:15.185322 (+ 16us) op_driver.cc:348] REPLICATION: starting
0114 02:54:15.458605 (+273283us) log.cc:844] Serialized 405 byte log entry
0114 02:54:15.459290 (+ 685us) spinlock_profiling.cc:243] Waited 273 ms on lock 0x7b54000316d4. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616ca6d1 00007f6461741198 00007f6461bad3b1 00007f6461bacb88 00007f6461bb1821 00007f6461bb17d9 00007f6461bb1769 00007f6461bb1731 00007f6461bb095d 000000000059ea04 000000000059e839
0114 02:54:15.645667 (+186377us) op_driver.cc:464] REPLICATION: finished
0114 02:54:15.646750 (+ 1083us) write_op.cc:301] APPLY: starting
0114 02:54:15.646804 (+ 54us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:15.646895 (+ 91us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:15.646912 (+ 17us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:15.647085 (+ 173us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:15.647220 (+ 135us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:15.647261 (+ 41us) write_op.cc:312] APPLY: finished
0114 02:54:15.647423 (+ 162us) log.cc:844] Serialized 23 byte log entry
0114 02:54:15.647573 (+ 150us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:15.647628 (+ 55us) write_op.cc:454] Released schema lock
0114 02:54:15.647658 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:15.647883 (+ 225us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:15.647910 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":942,"apply.run_cpu_time_us":1081,"apply.run_wall_time_us":1081,"num_ops":1,"prepare.queue_time_us":660020,"prepare.run_cpu_time_us":4501,"prepare.run_wall_time_us":275616,"replication_time_us":460252,"spinlock_wait_cycles":627292544}]]}
I20250114 02:54:15.679359 1206 ts_tablet_manager.cc:1428] T 96b06fe4173343dabd726d253b9f9b5f P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.039s user 0.003s sys 0.001s
I20250114 02:54:15.681924 1548 consensus_queue.cc:1035] T 268a12df225e4ba59d4a84c2869d4661 P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:15.687640 818 raft_consensus.cc:3054] T ed2d137579b8467da5eed4ea2f619a9a P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:15.691787 1205 raft_consensus.cc:357] T 290ad07ab8be4791a39194909e000843 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:15.694480 1205 raft_consensus.cc:383] T 290ad07ab8be4791a39194909e000843 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:15.695114 1205 raft_consensus.cc:738] T 290ad07ab8be4791a39194909e000843 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:15.695892 1206 tablet_bootstrap.cc:492] T e3af8f1299f24b08b2d25811942aaecc P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:15.696589 1508 raft_consensus.cc:491] T 15704785f56e43a3af69f181c75877ce P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 02:54:15.702145 469 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 961b6a25a13a4fa2a732a6383dcd8c84 consensus state has no leader
I20250114 02:54:15.703073 469 master_service.cc:946] ReplaceTablet: received request to replace tablet 96b06fe4173343dabd726d253b9f9b5f from {username='slave'} at 127.0.0.1:33540
I20250114 02:54:15.708298 470 master_service.cc:946] ReplaceTablet: received request to replace tablet db24bf3db61941ba9b5e0662db23e748 from {username='slave'} at 127.0.0.1:33540
I20250114 02:54:15.710633 1206 tablet_bootstrap.cc:654] T e3af8f1299f24b08b2d25811942aaecc P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:15.697091 1508 raft_consensus.cc:513] T 15704785f56e43a3af69f181c75877ce P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:15.739770 1508 leader_election.cc:290] T 15704785f56e43a3af69f181c75877ce P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:15.745894 959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15704785f56e43a3af69f181c75877ce" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:15.746299 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15704785f56e43a3af69f181c75877ce" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
W20250114 02:54:15.748273 1040 leader_election.cc:343] T 15704785f56e43a3af69f181c75877ce P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:15.747773 1508 consensus_queue.cc:1035] T ed2d137579b8467da5eed4ea2f619a9a P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
W20250114 02:54:15.751574 1042 leader_election.cc:343] T 15704785f56e43a3af69f181c75877ce P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:15.751960 1042 leader_election.cc:304] T 15704785f56e43a3af69f181c75877ce P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90d6c2c58bb54e0eafb365625ac33871; no voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2
I20250114 02:54:15.757277 1506 raft_consensus.cc:2743] T 15704785f56e43a3af69f181c75877ce P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:15.762897 1206 tablet_bootstrap.cc:492] T e3af8f1299f24b08b2d25811942aaecc P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:15.763303 1206 ts_tablet_manager.cc:1397] T e3af8f1299f24b08b2d25811942aaecc P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.068s user 0.013s sys 0.000s
I20250114 02:54:15.765686 1206 raft_consensus.cc:357] T e3af8f1299f24b08b2d25811942aaecc P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:15.766405 1206 raft_consensus.cc:383] T e3af8f1299f24b08b2d25811942aaecc P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:15.766638 1206 raft_consensus.cc:738] T e3af8f1299f24b08b2d25811942aaecc P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:15.767141 1206 consensus_queue.cc:260] T e3af8f1299f24b08b2d25811942aaecc P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:15.773504 1386 raft_consensus.cc:513] T e3af8f1299f24b08b2d25811942aaecc P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:15.776703 1512 consensus_queue.cc:1035] T 83b5b36a26e74668998a19fa0ddf7188 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:15.792049 464 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet 179248cb5acf4740b268e0ec8b2852f9 (table default.Table_bd405598fe054b718a8d6fb5dd875f0e [id=5745501ecd7b4cecaca3d35502a6cc40]). Expected version 1 got 0
I20250114 02:54:15.800189 817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3af8f1299f24b08b2d25811942aaecc" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:15.800809 817 raft_consensus.cc:3054] T e3af8f1299f24b08b2d25811942aaecc P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:15.800791 959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3af8f1299f24b08b2d25811942aaecc" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
I20250114 02:54:15.801378 959 raft_consensus.cc:3054] T e3af8f1299f24b08b2d25811942aaecc P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:15.807247 465 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 8b4bc535c42e43889401f14b6d59659e (table default.Table_d90b2995b13b4a8fb7e50cc4b79cbf84 [id=f8e40e0aa44b4157ac3438d11685b2e7]). Expected version 1 got 0
I20250114 02:54:15.807879 959 raft_consensus.cc:2463] T e3af8f1299f24b08b2d25811942aaecc P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1.
I20250114 02:54:15.809063 465 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet b63dd90f60a04465b948171e74888e79 (table default.Table_1097f09fb10e441fb639f8d7aa12d487 [id=d8c9691818d042d7b49475fa009c4ca1]). Expected version 3 got 0
I20250114 02:54:15.808981 1040 leader_election.cc:304] T e3af8f1299f24b08b2d25811942aaecc P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:15.809737 1508 raft_consensus.cc:2798] T e3af8f1299f24b08b2d25811942aaecc P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:15.809737 465 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 4bb9b429af92446bbe13894d5dfe4c47 (table default.Table_acb5bf9354be440bbf6034cf1c535dab [id=22b7edcf4b7e4c31b3c23dc44ccf028e]). Expected version 1 got 0
I20250114 02:54:15.810143 1508 raft_consensus.cc:695] T e3af8f1299f24b08b2d25811942aaecc P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Becoming Leader. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Running, Role: LEADER
I20250114 02:54:15.810848 1508 consensus_queue.cc:237] T e3af8f1299f24b08b2d25811942aaecc P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:15.820999 1206 ts_tablet_manager.cc:1428] T e3af8f1299f24b08b2d25811942aaecc P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.057s user 0.005s sys 0.004s
I20250114 02:54:15.821877 1386 leader_election.cc:290] T e3af8f1299f24b08b2d25811942aaecc P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Requested vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:15.822607 817 raft_consensus.cc:2463] T e3af8f1299f24b08b2d25811942aaecc P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1.
I20250114 02:54:15.697474 1205 consensus_queue.cc:260] T 290ad07ab8be4791a39194909e000843 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:15.810942 465 catalog_manager.cc:5526] T 25e85b0c5c9a4a3ca35a001359f21167 P 525198593c8e47afaee54e74267b87e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 525198593c8e47afaee54e74267b87e3 (127.0.107.66). New cstate: current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:15.828501 462 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet b4740f6b3a6741368ba68907ad065279
I20250114 02:54:15.829967 462 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:33540 (request call id 145) took 1711 ms. Trace:
I20250114 02:54:15.829871 1206 tablet_bootstrap.cc:492] T a3fdab330d574c0283ac58bc9e450b87 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:15.837177 940 tablet_service.cc:1514] Processing DeleteTablet for tablet b4740f6b3a6741368ba68907ad065279 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:14 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:15.838379 791 tablet_service.cc:1514] Processing DeleteTablet for tablet b4740f6b3a6741368ba68907ad065279 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:14 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:15.838785 1085 tablet_service.cc:1514] Processing DeleteTablet for tablet b4740f6b3a6741368ba68907ad065279 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:14 UTC) from {username='slave'} at 127.0.0.1:53876
W20250114 02:54:15.839362 466 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.65:49805 (request call id 40) took 1535 ms (client timeout 1000 ms). Trace:
I20250114 02:54:15.840425 794 tablet_service.cc:1514] Processing DeleteTablet for tablet 268a12df225e4ba59d4a84c2869d4661 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:10 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:15.840543 790 tablet_service.cc:1514] Processing DeleteTablet for tablet 83b5b36a26e74668998a19fa0ddf7188 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:12 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:15.830284 462 rpcz_store.cc:276] 0114 02:54:14.118315 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:14.122437 (+ 4122us) service_pool.cc:225] Handling call
0114 02:54:14.123972 (+ 1535us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0114 02:54:14.124757 (+ 785us) catalog_manager.cc:2753] Modifying in-memory table state
0114 02:54:14.124919 (+ 162us) catalog_manager.cc:2762] Locking tablets
0114 02:54:14.125628 (+ 709us) catalog_manager.cc:2776] Removing table and tablets from system table
0114 02:54:14.471807 (+346179us) catalog_manager.cc:2794] Removing table from by-name map
0114 02:54:14.472084 (+ 277us) catalog_manager.cc:2810] Committing in-memory state
0114 02:54:15.828313 (+1356229us) catalog_manager.cc:2814] Aborting table tasks
0114 02:54:15.829905 (+ 1592us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:14.381188 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:14.381535 (+ 347us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:14.381559 (+ 24us) write_op.cc:435] Acquired schema lock
0114 02:54:14.381571 (+ 12us) tablet.cc:659] Decoding operations
0114 02:54:14.381968 (+ 397us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:14.382008 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:14.382038 (+ 30us) tablet.cc:682] Acquiring locks for 2 operations
0114 02:54:14.382652 (+ 614us) tablet.cc:698] Row locks acquired
0114 02:54:14.382665 (+ 13us) write_op.cc:260] PREPARE: finished
0114 02:54:14.382763 (+ 98us) write_op.cc:270] Start()
0114 02:54:14.382846 (+ 83us) write_op.cc:276] Timestamp: P: 1736823254382727 usec, L: 0
0114 02:54:14.382862 (+ 16us) op_driver.cc:348] REPLICATION: starting
0114 02:54:14.431395 (+ 48533us) log.cc:844] Serialized 762 byte log entry
0114 02:54:14.469286 (+ 37891us) op_driver.cc:464] REPLICATION: finished
0114 02:54:14.470159 (+ 873us) write_op.cc:301] APPLY: starting
0114 02:54:14.470222 (+ 63us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:14.470333 (+ 111us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:14.470463 (+ 130us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:14.470691 (+ 228us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:14.470823 (+ 132us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0114 02:54:14.470862 (+ 39us) write_op.cc:312] APPLY: finished
0114 02:54:14.471049 (+ 187us) log.cc:844] Serialized 29 byte log entry
0114 02:54:14.471185 (+ 136us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:14.471246 (+ 61us) write_op.cc:454] Released schema lock
0114 02:54:14.471275 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:14.471482 (+ 207us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:14.471510 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":205,"child_traces":[["op",{"apply.queue_time_us":676,"apply.run_cpu_time_us":1302,"apply.run_wall_time_us":1302,"num_ops":2,"prepare.queue_time_us":253067,"prepare.run_cpu_time_us":2168,"prepare.run_wall_time_us":54556,"raft.queue_time_us":6694,"raft.run_cpu_time_us":667,"raft.run_wall_time_us":667,"replication_time_us":86325}]]}
I20250114 02:54:15.847635 462 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_dc087fb027f74c4a8bd5dede1142d97d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:15.851490 462 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dc087fb027f74c4a8bd5dede1142d97d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250114 02:54:15.839761 466 rpcz_store.cc:269] 0114 02:54:14.303371 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:14.303505 (+ 134us) service_pool.cc:225] Handling call
0114 02:54:15.839168 (+1535663us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:15.459531 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:15.460007 (+ 476us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:15.460059 (+ 52us) write_op.cc:435] Acquired schema lock
0114 02:54:15.460072 (+ 13us) tablet.cc:659] Decoding operations
0114 02:54:15.460418 (+ 346us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:15.460463 (+ 45us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:15.460496 (+ 33us) tablet.cc:682] Acquiring locks for 1 operations
0114 02:54:15.460672 (+ 176us) tablet.cc:698] Row locks acquired
0114 02:54:15.460707 (+ 35us) write_op.cc:260] PREPARE: finished
0114 02:54:15.460841 (+ 134us) write_op.cc:270] Start()
0114 02:54:15.460912 (+ 71us) write_op.cc:276] Timestamp: P: 1736823255460697 usec, L: 0
0114 02:54:15.460931 (+ 19us) op_driver.cc:348] REPLICATION: starting
0114 02:54:15.461533 (+ 602us) log.cc:844] Serialized 530 byte log entry
0114 02:54:15.645897 (+184364us) op_driver.cc:464] REPLICATION: finished
0114 02:54:15.648124 (+ 2227us) write_op.cc:301] APPLY: starting
0114 02:54:15.648170 (+ 46us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:15.648279 (+ 109us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:15.648372 (+ 93us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:15.648520 (+ 148us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:15.648665 (+ 145us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:15.648701 (+ 36us) write_op.cc:312] APPLY: finished
0114 02:54:15.648841 (+ 140us) log.cc:844] Serialized 23 byte log entry
0114 02:54:15.648952 (+ 111us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:15.648998 (+ 46us) write_op.cc:454] Released schema lock
0114 02:54:15.649028 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:15.649224 (+ 196us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:15.649245 (+ 21us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":7,"child_traces":[["op",{"apply.queue_time_us":2090,"apply.run_cpu_time_us":1062,"apply.run_wall_time_us":1057,"num_ops":1,"prepare.queue_time_us":916822,"prepare.run_cpu_time_us":2198,"prepare.run_wall_time_us":2198,"replication_time_us":184875}]]}
I20250114 02:54:15.853313 466 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_6bdcd457ba84407e8ec4b0fb6a1d04c0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:15.855003 466 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6bdcd457ba84407e8ec4b0fb6a1d04c0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250114 02:54:15.857386 455 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.125:46837, user_credentials={real_user=slave}} blocked reactor thread for 95869.6us
I20250114 02:54:15.861423 936 tablet_service.cc:1514] Processing DeleteTablet for tablet 83b5b36a26e74668998a19fa0ddf7188 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:12 UTC) from {username='slave'} at 127.0.0.1:55068
W20250114 02:54:15.862277 463 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.66:50663 (request call id 38) took 1508 ms (client timeout 1000 ms). Trace:
I20250114 02:54:15.862453 936 tablet_service.cc:1514] Processing DeleteTablet for tablet 268a12df225e4ba59d4a84c2869d4661 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:10 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:15.866160 468 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet 276942e5e54246c79f0d4f08297f58ea
I20250114 02:54:15.867461 468 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:33540 (request call id 149) took 1596 ms. Trace:
I20250114 02:54:15.869122 936 tablet_service.cc:1514] Processing DeleteTablet for tablet 276942e5e54246c79f0d4f08297f58ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:14 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:15.871299 1205 ts_tablet_manager.cc:1428] T 290ad07ab8be4791a39194909e000843 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.234s user 0.006s sys 0.000s
I20250114 02:54:15.867722 468 rpcz_store.cc:276] 0114 02:54:14.271578 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:14.271700 (+ 122us) service_pool.cc:225] Handling call
0114 02:54:14.297843 (+ 26143us) spinlock_profiling.cc:243] Waited 25.3 ms on lock 0x7b54000316d4. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616cad50 00007f646174df2a 00007f64633f2dec 00007f64634ec616 00007f6461254b44 00007f6461254ad2 00007f6461254a01 00007f646125497c 00007f6461253c32 00007f645f512dfc 00007f645f512236
0114 02:54:14.298871 (+ 1028us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0114 02:54:14.299398 (+ 527us) catalog_manager.cc:2753] Modifying in-memory table state
0114 02:54:14.299527 (+ 129us) catalog_manager.cc:2762] Locking tablets
0114 02:54:14.508273 (+208746us) catalog_manager.cc:2776] Removing table and tablets from system table
0114 02:54:15.462848 (+954575us) catalog_manager.cc:2794] Removing table from by-name map
0114 02:54:15.463199 (+ 351us) catalog_manager.cc:2810] Committing in-memory state
0114 02:54:15.865970 (+402771us) catalog_manager.cc:2814] Aborting table tasks
0114 02:54:15.867408 (+ 1438us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:15.078578 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:15.079036 (+ 458us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:15.079065 (+ 29us) write_op.cc:435] Acquired schema lock
0114 02:54:15.079083 (+ 18us) tablet.cc:659] Decoding operations
0114 02:54:15.079617 (+ 534us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:15.079661 (+ 44us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:15.079688 (+ 27us) tablet.cc:682] Acquiring locks for 2 operations
0114 02:54:15.079977 (+ 289us) tablet.cc:698] Row locks acquired
0114 02:54:15.079993 (+ 16us) write_op.cc:260] PREPARE: finished
0114 02:54:15.080092 (+ 99us) write_op.cc:270] Start()
0114 02:54:15.080171 (+ 79us) write_op.cc:276] Timestamp: P: 1736823255079986 usec, L: 0
0114 02:54:15.080187 (+ 16us) op_driver.cc:348] REPLICATION: starting
0114 02:54:15.144653 (+ 64466us) log.cc:844] Serialized 759 byte log entry
0114 02:54:15.457808 (+313155us) op_driver.cc:464] REPLICATION: finished
0114 02:54:15.461452 (+ 3644us) write_op.cc:301] APPLY: starting
0114 02:54:15.461511 (+ 59us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:15.461613 (+ 102us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:15.461698 (+ 85us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:15.462027 (+ 329us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:15.462196 (+ 169us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0114 02:54:15.462232 (+ 36us) write_op.cc:312] APPLY: finished
0114 02:54:15.462403 (+ 171us) log.cc:844] Serialized 29 byte log entry
0114 02:54:15.462513 (+ 110us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:15.462564 (+ 51us) write_op.cc:454] Released schema lock
0114 02:54:15.462589 (+ 25us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:15.462825 (+ 236us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:15.462859 (+ 34us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":58269568,"child_traces":[["op",{"apply.queue_time_us":3477,"apply.run_cpu_time_us":1329,"apply.run_wall_time_us":1330,"num_ops":2,"prepare.queue_time_us":568561,"prepare.run_cpu_time_us":2795,"prepare.run_wall_time_us":105066,"raft.queue_time_us":111,"raft.run_cpu_time_us":967,"raft.run_wall_time_us":967,"replication_time_us":377542}]]}
I20250114 02:54:15.874439 1083 tablet_service.cc:1514] Processing DeleteTablet for tablet 276942e5e54246c79f0d4f08297f58ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:14 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:15.878765 1083 tablet_service.cc:1514] Processing DeleteTablet for tablet 268a12df225e4ba59d4a84c2869d4661 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:10 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:15.879878 1083 tablet_service.cc:1514] Processing DeleteTablet for tablet ed2d137579b8467da5eed4ea2f619a9a with delete_type TABLET_DATA_DELETED (replaced by tablet 0304df55431e42e1a999ed17dc35e6c0) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:15.882174 1083 tablet_service.cc:1514] Processing DeleteTablet for tablet 9589afd0094545c1801b7a0b5eb8698f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:13 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:15.884562 790 tablet_service.cc:1514] Processing DeleteTablet for tablet 9589afd0094545c1801b7a0b5eb8698f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:13 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:15.885610 790 tablet_service.cc:1514] Processing DeleteTablet for tablet 276942e5e54246c79f0d4f08297f58ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:14 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:15.887881 1593 tablet_replica.cc:331] T b4740f6b3a6741368ba68907ad065279 P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:15.889181 1594 tablet_replica.cc:331] T b4740f6b3a6741368ba68907ad065279 P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:15.883562 1083 tablet_service.cc:1514] Processing DeleteTablet for tablet c20aca761313498193e1d79c10705f8f with delete_type TABLET_DATA_DELETED (replaced by tablet bc46815b4803418c95ea9c039a4fae88) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:15.892959 1593 raft_consensus.cc:2238] T b4740f6b3a6741368ba68907ad065279 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:15.893581 1593 raft_consensus.cc:2267] T b4740f6b3a6741368ba68907ad065279 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:15.893189 468 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_23120ced31c744a0ba995f46d50223a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:15.895351 468 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_23120ced31c744a0ba995f46d50223a3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:15.896505 1594 raft_consensus.cc:2238] T b4740f6b3a6741368ba68907ad065279 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 LEADER]: Raft consensus shutting down.
I20250114 02:54:15.897518 1594 raft_consensus.cc:2267] T b4740f6b3a6741368ba68907ad065279 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:15.901635 1594 ts_tablet_manager.cc:1905] T b4740f6b3a6741368ba68907ad065279 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:15.902261 1207 tablet_bootstrap.cc:492] T 7ecf782bb3d54528aa996abca1ea3a15 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:15.902657 1207 ts_tablet_manager.cc:1397] T 7ecf782bb3d54528aa996abca1ea3a15 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.305s user 0.017s sys 0.000s
I20250114 02:54:15.902796 1593 ts_tablet_manager.cc:1905] T b4740f6b3a6741368ba68907ad065279 P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:15.904588 1206 tablet_bootstrap.cc:654] T a3fdab330d574c0283ac58bc9e450b87 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:15.920621 470 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet db24bf3db61941ba9b5e0662db23e748
I20250114 02:54:15.922331 940 tablet_service.cc:1514] Processing DeleteTablet for tablet db24bf3db61941ba9b5e0662db23e748 with delete_type TABLET_DATA_DELETED (replaced by tablet ddfbab181b1f474192c664cd0316037f) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:15.922560 791 tablet_service.cc:1514] Processing DeleteTablet for tablet db24bf3db61941ba9b5e0662db23e748 with delete_type TABLET_DATA_DELETED (replaced by tablet ddfbab181b1f474192c664cd0316037f) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:15.923839 470 catalog_manager.cc:6596] ReplaceTablet: tablet db24bf3db61941ba9b5e0662db23e748 deleted and replaced by tablet ddfbab181b1f474192c664cd0316037f
I20250114 02:54:15.924804 1083 tablet_service.cc:1514] Processing DeleteTablet for tablet db24bf3db61941ba9b5e0662db23e748 with delete_type TABLET_DATA_DELETED (replaced by tablet ddfbab181b1f474192c664cd0316037f) from {username='slave'} at 127.0.0.1:53876
W20250114 02:54:15.933290 1582 log.cc:926] Time spent T 268a12df225e4ba59d4a84c2869d4661 P 525198593c8e47afaee54e74267b87e3: Append to log took a long time: real 0.235s user 0.000s sys 0.000s
I20250114 02:54:15.934398 1205 tablet_bootstrap.cc:492] T 15704785f56e43a3af69f181c75877ce P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:15.937206 1512 raft_consensus.cc:491] T 290ad07ab8be4791a39194909e000843 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:15.937646 1512 raft_consensus.cc:513] T 290ad07ab8be4791a39194909e000843 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:15.939471 1512 leader_election.cc:290] T 290ad07ab8be4791a39194909e000843 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:15.946081 1595 tablet_replica.cc:331] T b4740f6b3a6741368ba68907ad065279 P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:15.954228 1598 tablet.cc:1721] T d434cbc9daf549a6bdc04c8c76ead8a6 P 819f9e3412e54d44abb11d8b3ca877c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250114 02:54:15.980026 464 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.67:34825 (request call id 37) took 1616 ms (client timeout 1000 ms). Trace:
I20250114 02:54:15.991725 1593 ts_tablet_manager.cc:1918] T b4740f6b3a6741368ba68907ad065279 P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:15.992113 1593 log.cc:1198] T b4740f6b3a6741368ba68907ad065279 P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/b4740f6b3a6741368ba68907ad065279
I20250114 02:54:15.949229 1207 raft_consensus.cc:357] T 7ecf782bb3d54528aa996abca1ea3a15 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:15.993566 1207 raft_consensus.cc:383] T 7ecf782bb3d54528aa996abca1ea3a15 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:15.993844 1207 raft_consensus.cc:738] T 7ecf782bb3d54528aa996abca1ea3a15 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:15.994403 1207 consensus_queue.cc:260] T 7ecf782bb3d54528aa996abca1ea3a15 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
W20250114 02:54:15.988097 464 rpcz_store.cc:269] 0114 02:54:14.363689 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:14.363811 (+ 122us) service_pool.cc:225] Handling call
0114 02:54:15.972617 (+1608806us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":9}
I20250114 02:54:15.998371 1593 ts_tablet_manager.cc:1939] T b4740f6b3a6741368ba68907ad065279 P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:15.999365 1595 raft_consensus.cc:2238] T b4740f6b3a6741368ba68907ad065279 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:16.005519 1205 tablet_bootstrap.cc:654] T 15704785f56e43a3af69f181c75877ce P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:16.009284 1593 tablet_replica.cc:331] T 83b5b36a26e74668998a19fa0ddf7188 P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:16.013839 1593 raft_consensus.cc:2238] T 83b5b36a26e74668998a19fa0ddf7188 P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Raft consensus shutting down.
I20250114 02:54:16.001914 469 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet 96b06fe4173343dabd726d253b9f9b5f
I20250114 02:54:16.003585 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet b4740f6b3a6741368ba68907ad065279 (table default.Table_8a880e0ae5ff4b5d8de8f40f15dd6e02 [id=6baaeac9e83d4e3092485fca13e07303]) successfully deleted
I20250114 02:54:16.012776 470 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_bd405598fe054b718a8d6fb5dd875f0e" } new_table_name: "default.Table_58e014a2cfee480184ff3c66b1768cf6" modify_external_catalogs: true
I20250114 02:54:16.029541 469 catalog_manager.cc:6596] ReplaceTablet: tablet 96b06fe4173343dabd726d253b9f9b5f deleted and replaced by tablet e15e6e52d7294aeeb4463b3a76022ec3
I20250114 02:54:16.033836 1597 raft_consensus.cc:491] T 179248cb5acf4740b268e0ec8b2852f9 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:16.034278 1597 raft_consensus.cc:513] T 179248cb5acf4740b268e0ec8b2852f9 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:16.035128 940 tablet_service.cc:1514] Processing DeleteTablet for tablet 96b06fe4173343dabd726d253b9f9b5f with delete_type TABLET_DATA_DELETED (replaced by tablet e15e6e52d7294aeeb4463b3a76022ec3) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:16.036154 791 tablet_service.cc:1514] Processing DeleteTablet for tablet 96b06fe4173343dabd726d253b9f9b5f with delete_type TABLET_DATA_DELETED (replaced by tablet e15e6e52d7294aeeb4463b3a76022ec3) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:16.036163 1597 leader_election.cc:290] T 179248cb5acf4740b268e0ec8b2852f9 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:16.038866 469 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_42c4106f7e6644d79bb958e314e449e5" } new_table_name: "default.Table_66f2263162f64250a0b49073abe4bd4c" modify_external_catalogs: true
I20250114 02:54:16.040350 1206 tablet_bootstrap.cc:492] T a3fdab330d574c0283ac58bc9e450b87 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:16.040757 1206 ts_tablet_manager.cc:1397] T a3fdab330d574c0283ac58bc9e450b87 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.214s user 0.021s sys 0.010s
I20250114 02:54:16.043118 464 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_3e7cad29b0c0447ab9539b45ee482232"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:16.042971 1206 raft_consensus.cc:357] T a3fdab330d574c0283ac58bc9e450b87 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:16.044833 1206 raft_consensus.cc:383] T a3fdab330d574c0283ac58bc9e450b87 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:16.045094 1206 raft_consensus.cc:738] T a3fdab330d574c0283ac58bc9e450b87 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:16.045754 1206 consensus_queue.cc:260] T a3fdab330d574c0283ac58bc9e450b87 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:16.049602 1206 ts_tablet_manager.cc:1428] T a3fdab330d574c0283ac58bc9e450b87 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.009s user 0.005s sys 0.001s
I20250114 02:54:16.050580 1206 tablet_bootstrap.cc:492] T 290ad07ab8be4791a39194909e000843 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:16.052660 1205 tablet_bootstrap.cc:492] T 15704785f56e43a3af69f181c75877ce P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:16.053071 1205 ts_tablet_manager.cc:1397] T 15704785f56e43a3af69f181c75877ce P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.119s user 0.009s sys 0.000s
I20250114 02:54:16.066227 1206 tablet_bootstrap.cc:654] T 290ad07ab8be4791a39194909e000843 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:16.068176 1205 raft_consensus.cc:357] T 15704785f56e43a3af69f181c75877ce P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:16.068940 1205 raft_consensus.cc:383] T 15704785f56e43a3af69f181c75877ce P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:16.069319 1205 raft_consensus.cc:738] T 15704785f56e43a3af69f181c75877ce P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:16.070009 1205 consensus_queue.cc:260] T 15704785f56e43a3af69f181c75877ce P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:16.081472 1602 tablet.cc:1721] T d434cbc9daf549a6bdc04c8c76ead8a6 P 525198593c8e47afaee54e74267b87e3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250114 02:54:15.999766 1508 consensus_queue.cc:1035] T ed2d137579b8467da5eed4ea2f619a9a P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:16.030596 1207 ts_tablet_manager.cc:1428] T 7ecf782bb3d54528aa996abca1ea3a15 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.128s user 0.008s sys 0.000s
I20250114 02:54:16.037135 1085 tablet_service.cc:1514] Processing DeleteTablet for tablet 96b06fe4173343dabd726d253b9f9b5f with delete_type TABLET_DATA_DELETED (replaced by tablet e15e6e52d7294aeeb4463b3a76022ec3) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:16.092950 1207 tablet_bootstrap.cc:492] T 00080bea6c8e46e584ccba6dcb0e221b P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:16.114014 1207 tablet_bootstrap.cc:654] T 00080bea6c8e46e584ccba6dcb0e221b P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:15.984714 1596 tablet.cc:1721] T d434cbc9daf549a6bdc04c8c76ead8a6 P 90d6c2c58bb54e0eafb365625ac33871: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250114 02:54:16.142082 1206 tablet_bootstrap.cc:492] T 290ad07ab8be4791a39194909e000843 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:16.142488 1206 ts_tablet_manager.cc:1397] T 290ad07ab8be4791a39194909e000843 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.092s user 0.019s sys 0.003s
I20250114 02:54:16.144707 1206 raft_consensus.cc:357] T 290ad07ab8be4791a39194909e000843 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:16.145372 1206 raft_consensus.cc:383] T 290ad07ab8be4791a39194909e000843 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:16.145627 1206 raft_consensus.cc:738] T 290ad07ab8be4791a39194909e000843 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:16.148699 1206 consensus_queue.cc:260] T 290ad07ab8be4791a39194909e000843 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:16.155011 1206 ts_tablet_manager.cc:1428] T 290ad07ab8be4791a39194909e000843 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.012s user 0.006s sys 0.001s
I20250114 02:54:16.155807 1206 tablet_bootstrap.cc:492] T 7ecf782bb3d54528aa996abca1ea3a15 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
W20250114 02:54:16.178870 464 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3e7cad29b0c0447ab9539b45ee482232 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250114 02:54:16.084817 898 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.65:39397, user_credentials={real_user=slave}} blocked reactor thread for 147906us
I20250114 02:54:16.090200 1205 ts_tablet_manager.cc:1428] T 15704785f56e43a3af69f181c75877ce P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.025s user 0.002s sys 0.004s
I20250114 02:54:16.197156 1205 tablet_bootstrap.cc:492] T 7ecf782bb3d54528aa996abca1ea3a15 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:16.197623 816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "290ad07ab8be4791a39194909e000843" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:16.198233 816 raft_consensus.cc:2463] T 290ad07ab8be4791a39194909e000843 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:16.202265 1205 tablet_bootstrap.cc:654] T 7ecf782bb3d54528aa996abca1ea3a15 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:16.203685 961 raft_consensus.cc:3054] T 179248cb5acf4740b268e0ec8b2852f9 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:16.090757 1593 raft_consensus.cc:2267] T 83b5b36a26e74668998a19fa0ddf7188 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:16.152818 1512 raft_consensus.cc:491] T a3fdab330d574c0283ac58bc9e450b87 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:16.205067 1512 raft_consensus.cc:513] T a3fdab330d574c0283ac58bc9e450b87 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:16.206760 1512 leader_election.cc:290] T a3fdab330d574c0283ac58bc9e450b87 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:16.211001 1593 ts_tablet_manager.cc:1905] T 83b5b36a26e74668998a19fa0ddf7188 P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:16.214061 898 leader_election.cc:304] T 290ad07ab8be4791a39194909e000843 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
I20250114 02:54:16.215799 1601 raft_consensus.cc:2798] T 290ad07ab8be4791a39194909e000843 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:16.216162 1601 raft_consensus.cc:491] T 290ad07ab8be4791a39194909e000843 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:16.216543 1601 raft_consensus.cc:3054] T 290ad07ab8be4791a39194909e000843 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:16.223649 1601 raft_consensus.cc:513] T 290ad07ab8be4791a39194909e000843 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:16.225490 1601 leader_election.cc:290] T 290ad07ab8be4791a39194909e000843 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Requested vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:16.236048 1294 consensus_queue.cc:1035] T 179248cb5acf4740b268e0ec8b2852f9 P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:54:16.239413 815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "179248cb5acf4740b268e0ec8b2852f9" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:16.243538 1512 raft_consensus.cc:491] T 9bfd8fd6d2104fb4a6484cb8c98981bd P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:16.244045 1512 raft_consensus.cc:513] T 9bfd8fd6d2104fb4a6484cb8c98981bd P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:16.246745 1512 leader_election.cc:290] T 9bfd8fd6d2104fb4a6484cb8c98981bd P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:16.250605 814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3fdab330d574c0283ac58bc9e450b87" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:16.252063 814 raft_consensus.cc:2463] T a3fdab330d574c0283ac58bc9e450b87 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:16.254011 813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "290ad07ab8be4791a39194909e000843" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:16.254701 813 raft_consensus.cc:3054] T 290ad07ab8be4791a39194909e000843 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:16.253185 1600 consensus_queue.cc:1035] T 25e85b0c5c9a4a3ca35a001359f21167 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:54:16.267136 815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bfd8fd6d2104fb4a6484cb8c98981bd" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:16.267835 815 raft_consensus.cc:2463] T 9bfd8fd6d2104fb4a6484cb8c98981bd P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:16.277976 1205 tablet_bootstrap.cc:492] T 7ecf782bb3d54528aa996abca1ea3a15 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:16.278792 1205 ts_tablet_manager.cc:1397] T 7ecf782bb3d54528aa996abca1ea3a15 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.082s user 0.012s sys 0.000s
I20250114 02:54:16.274734 898 leader_election.cc:304] T a3fdab330d574c0283ac58bc9e450b87 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
I20250114 02:54:16.281401 1601 raft_consensus.cc:2798] T a3fdab330d574c0283ac58bc9e450b87 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:16.281764 1601 raft_consensus.cc:491] T a3fdab330d574c0283ac58bc9e450b87 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250114 02:54:15.862625 463 rpcz_store.cc:269] 0114 02:54:14.354096 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:14.359317 (+ 5221us) service_pool.cc:225] Handling call
0114 02:54:15.862208 (+1502891us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:15.686898 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:15.687274 (+ 376us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:15.687296 (+ 22us) write_op.cc:435] Acquired schema lock
0114 02:54:15.687309 (+ 13us) tablet.cc:659] Decoding operations
0114 02:54:15.687557 (+ 248us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:15.687589 (+ 32us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:15.687609 (+ 20us) tablet.cc:682] Acquiring locks for 1 operations
0114 02:54:15.687742 (+ 133us) tablet.cc:698] Row locks acquired
0114 02:54:15.687754 (+ 12us) write_op.cc:260] PREPARE: finished
0114 02:54:15.687898 (+ 144us) write_op.cc:270] Start()
0114 02:54:15.687973 (+ 75us) write_op.cc:276] Timestamp: P: 1736823255687773 usec, L: 0
0114 02:54:15.687990 (+ 17us) op_driver.cc:348] REPLICATION: starting
0114 02:54:15.711146 (+ 23156us) log.cc:844] Serialized 530 byte log entry
0114 02:54:15.711692 (+ 546us) spinlock_profiling.cc:243] Waited 22.5 ms on lock 0x7b54000316d4. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616ca6d1 00007f6461741198 00007f6461bad3b1 00007f6461bacb88 00007f6461bb1821 00007f6461bb17d9 00007f6461bb1769 00007f6461bb1731 00007f6461bb095d 000000000059ea04 000000000059e839
0114 02:54:15.722470 (+ 10778us) op_driver.cc:464] REPLICATION: finished
0114 02:54:15.742832 (+ 20362us) write_op.cc:301] APPLY: starting
0114 02:54:15.742888 (+ 56us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:15.742978 (+ 90us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:15.743004 (+ 26us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:15.743172 (+ 168us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:15.743304 (+ 132us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:15.743347 (+ 43us) write_op.cc:312] APPLY: finished
0114 02:54:15.743521 (+ 174us) log.cc:844] Serialized 23 byte log entry
0114 02:54:15.744072 (+ 551us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:15.744132 (+ 60us) write_op.cc:454] Released schema lock
0114 02:54:15.744160 (+ 28us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:15.744418 (+ 258us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:15.744448 (+ 30us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":5,"child_traces":[["op",{"apply.queue_time_us":20039,"apply.run_cpu_time_us":1173,"apply.run_wall_time_us":1558,"num_ops":1,"prepare.queue_time_us":350,"prepare.run_cpu_time_us":2742,"prepare.run_wall_time_us":25029,"raft.queue_time_us":480,"raft.run_cpu_time_us":544,"raft.run_wall_time_us":557,"replication_time_us":34401,"spinlock_wait_cycles":51683200}]]}
I20250114 02:54:16.282269 1601 raft_consensus.cc:3054] T a3fdab330d574c0283ac58bc9e450b87 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
W20250114 02:54:16.226560 455 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.125:46837, user_credentials={real_user=slave}} blocked reactor thread for 63924.6us
I20250114 02:54:16.283120 463 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_661ed49bcbbb476aa68742b72604d95f" } modify_external_catalogs: true
I20250114 02:54:16.230700 466 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_b0cd93e5a498450e84fb0f1ba4be4505" } modify_external_catalogs: true
I20250114 02:54:16.283685 463 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_661ed49bcbbb476aa68742b72604d95f" } modify_external_catalogs: true
I20250114 02:54:16.283813 466 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_b0cd93e5a498450e84fb0f1ba4be4505" } modify_external_catalogs: true
I20250114 02:54:16.289631 898 leader_election.cc:304] T 9bfd8fd6d2104fb4a6484cb8c98981bd P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
I20250114 02:54:16.298713 1593 ts_tablet_manager.cc:1918] T 83b5b36a26e74668998a19fa0ddf7188 P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:16.299166 1593 log.cc:1198] T 83b5b36a26e74668998a19fa0ddf7188 P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/83b5b36a26e74668998a19fa0ddf7188
I20250114 02:54:16.299919 1593 ts_tablet_manager.cc:1939] T 83b5b36a26e74668998a19fa0ddf7188 P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:16.300504 1600 raft_consensus.cc:2798] T 9bfd8fd6d2104fb4a6484cb8c98981bd P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:16.300840 1600 raft_consensus.cc:491] T 9bfd8fd6d2104fb4a6484cb8c98981bd P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:16.301154 1600 raft_consensus.cc:3054] T 9bfd8fd6d2104fb4a6484cb8c98981bd P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:16.312685 1600 raft_consensus.cc:513] T 9bfd8fd6d2104fb4a6484cb8c98981bd P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
W20250114 02:54:16.319720 1596 tablet_replica.cc:1411] Time spent flushing the Log queue took a long time: real 0.203s user 0.000s sys 0.001s
I20250114 02:54:16.314761 1506 raft_consensus.cc:491] T 7ecf782bb3d54528aa996abca1ea3a15 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:16.294024 813 raft_consensus.cc:2463] T 290ad07ab8be4791a39194909e000843 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:16.332132 1600 leader_election.cc:290] T 9bfd8fd6d2104fb4a6484cb8c98981bd P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Requested vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:16.329792 1205 raft_consensus.cc:357] T 7ecf782bb3d54528aa996abca1ea3a15 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:16.340973 1205 raft_consensus.cc:383] T 7ecf782bb3d54528aa996abca1ea3a15 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:16.341305 1205 raft_consensus.cc:738] T 7ecf782bb3d54528aa996abca1ea3a15 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:16.341526 1601 raft_consensus.cc:513] T a3fdab330d574c0283ac58bc9e450b87 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:16.341974 1205 consensus_queue.cc:260] T 7ecf782bb3d54528aa996abca1ea3a15 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:16.343248 1601 leader_election.cc:290] T a3fdab330d574c0283ac58bc9e450b87 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Requested vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:16.344224 1205 ts_tablet_manager.cc:1428] T 7ecf782bb3d54528aa996abca1ea3a15 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.065s user 0.005s sys 0.001s
I20250114 02:54:16.349339 1598 tablet.cc:1721] T b63dd90f60a04465b948171e74888e79 P 819f9e3412e54d44abb11d8b3ca877c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250114 02:54:16.323439 1595 raft_consensus.cc:2267] T b4740f6b3a6741368ba68907ad065279 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:16.355010 815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3fdab330d574c0283ac58bc9e450b87" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:16.355583 815 raft_consensus.cc:3054] T a3fdab330d574c0283ac58bc9e450b87 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:16.321168 1100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "179248cb5acf4740b268e0ec8b2852f9" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:16.354103 1104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "290ad07ab8be4791a39194909e000843" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:16.326323 1506 raft_consensus.cc:513] T 7ecf782bb3d54528aa996abca1ea3a15 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:16.363655 1594 ts_tablet_manager.cc:1918] T b4740f6b3a6741368ba68907ad065279 P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:16.364745 1506 leader_election.cc:290] T 7ecf782bb3d54528aa996abca1ea3a15 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:16.366045 1616 tablet.cc:1721] T b63dd90f60a04465b948171e74888e79 P 525198593c8e47afaee54e74267b87e3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250114 02:54:16.366945 815 raft_consensus.cc:2463] T a3fdab330d574c0283ac58bc9e450b87 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:16.367796 1600 raft_consensus.cc:491] T 96b06fe4173343dabd726d253b9f9b5f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:16.368408 1600 raft_consensus.cc:513] T 96b06fe4173343dabd726d253b9f9b5f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:16.368760 898 leader_election.cc:304] T 290ad07ab8be4791a39194909e000843 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
I20250114 02:54:16.370451 1600 leader_election.cc:290] T 96b06fe4173343dabd726d253b9f9b5f P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:16.370494 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bfd8fd6d2104fb4a6484cb8c98981bd" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:16.370954 818 raft_consensus.cc:3054] T 9bfd8fd6d2104fb4a6484cb8c98981bd P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:16.370970 1600 raft_consensus.cc:2798] T 290ad07ab8be4791a39194909e000843 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:16.371424 1600 raft_consensus.cc:695] T 290ad07ab8be4791a39194909e000843 P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Becoming Leader. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Running, Role: LEADER
I20250114 02:54:16.372005 1098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "290ad07ab8be4791a39194909e000843" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:16.372179 1600 consensus_queue.cc:237] T 290ad07ab8be4791a39194909e000843 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:16.378644 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet 83b5b36a26e74668998a19fa0ddf7188 (table default.Table_acca64d3328d47578d6d1ebd594babbd [id=bb08f9cb85c44d108156675ff51b37c1]) successfully deleted
I20250114 02:54:16.380754 1100 raft_consensus.cc:2388] T 179248cb5acf4740b268e0ec8b2852f9 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 525198593c8e47afaee54e74267b87e3 in current term 1: Already voted for candidate 819f9e3412e54d44abb11d8b3ca877c2 in this term.
I20250114 02:54:16.364065 1594 log.cc:1198] T b4740f6b3a6741368ba68907ad065279 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/b4740f6b3a6741368ba68907ad065279
I20250114 02:54:16.385083 1594 ts_tablet_manager.cc:1939] T b4740f6b3a6741368ba68907ad065279 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:16.386370 1594 tablet_replica.cc:331] T 268a12df225e4ba59d4a84c2869d4661 P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:16.386600 1595 ts_tablet_manager.cc:1905] T b4740f6b3a6741368ba68907ad065279 P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:16.387291 1594 raft_consensus.cc:2238] T 268a12df225e4ba59d4a84c2869d4661 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 LEADER]: Raft consensus shutting down.
I20250114 02:54:16.376073 1593 tablet_replica.cc:331] T 268a12df225e4ba59d4a84c2869d4661 P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:16.389626 1614 tablet.cc:1721] T 8b4bc535c42e43889401f14b6d59659e P 525198593c8e47afaee54e74267b87e3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250114 02:54:16.389919 1593 raft_consensus.cc:2238] T 268a12df225e4ba59d4a84c2869d4661 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:16.397830 1593 raft_consensus.cc:2267] T 268a12df225e4ba59d4a84c2869d4661 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:16.367810 1104 raft_consensus.cc:2463] T 290ad07ab8be4791a39194909e000843 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:16.405997 1099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3fdab330d574c0283ac58bc9e450b87" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:16.406999 1099 raft_consensus.cc:2463] T a3fdab330d574c0283ac58bc9e450b87 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:16.413115 813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ecf782bb3d54528aa996abca1ea3a15" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:16.415486 961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ecf782bb3d54528aa996abca1ea3a15" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:16.416152 961 raft_consensus.cc:2463] T 7ecf782bb3d54528aa996abca1ea3a15 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 0.
I20250114 02:54:16.416958 1205 tablet_bootstrap.cc:492] T de4ca83a69704a368a25954cdc1b9392 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
W20250114 02:54:16.423126 898 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.65:39397, user_credentials={real_user=slave}} blocked reactor thread for 53621.5us
I20250114 02:54:16.424038 898 leader_election.cc:304] T a3fdab330d574c0283ac58bc9e450b87 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
I20250114 02:54:16.426420 1597 raft_consensus.cc:2798] T a3fdab330d574c0283ac58bc9e450b87 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:16.426889 1597 raft_consensus.cc:695] T a3fdab330d574c0283ac58bc9e450b87 P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Becoming Leader. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Running, Role: LEADER
I20250114 02:54:16.426882 1104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bfd8fd6d2104fb4a6484cb8c98981bd" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:16.427526 1104 raft_consensus.cc:3054] T 9bfd8fd6d2104fb4a6484cb8c98981bd P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:16.428640 1597 consensus_queue.cc:237] T a3fdab330d574c0283ac58bc9e450b87 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:16.436115 1548 consensus_queue.cc:1035] T 179248cb5acf4740b268e0ec8b2852f9 P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:16.421140 818 raft_consensus.cc:2463] T 9bfd8fd6d2104fb4a6484cb8c98981bd P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:16.442294 1040 leader_election.cc:304] T 7ecf782bb3d54528aa996abca1ea3a15 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:16.444377 1506 raft_consensus.cc:2798] T 7ecf782bb3d54528aa996abca1ea3a15 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:16.444693 1506 raft_consensus.cc:491] T 7ecf782bb3d54528aa996abca1ea3a15 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:16.444998 1506 raft_consensus.cc:3054] T 7ecf782bb3d54528aa996abca1ea3a15 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:16.453110 1594 raft_consensus.cc:2267] T 268a12df225e4ba59d4a84c2869d4661 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:16.457852 1594 ts_tablet_manager.cc:1905] T 268a12df225e4ba59d4a84c2869d4661 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:16.459071 1505 consensus_queue.cc:1035] T c20aca761313498193e1d79c10705f8f P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250114 02:54:16.460402 1042 leader_election.cc:343] T 7ecf782bb3d54528aa996abca1ea3a15 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:16.461858 1508 consensus_queue.cc:1035] T c20aca761313498193e1d79c10705f8f P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:16.462790 1205 tablet_bootstrap.cc:654] T de4ca83a69704a368a25954cdc1b9392 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:16.467772 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96b06fe4173343dabd726d253b9f9b5f" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:16.469054 818 raft_consensus.cc:2463] T 96b06fe4173343dabd726d253b9f9b5f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:16.473668 1598 tablet.cc:1721] T 8b4bc535c42e43889401f14b6d59659e P 819f9e3412e54d44abb11d8b3ca877c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250114 02:54:16.475872 1104 raft_consensus.cc:2463] T 9bfd8fd6d2104fb4a6484cb8c98981bd P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:16.476364 1103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3fdab330d574c0283ac58bc9e450b87" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:16.477985 1103 raft_consensus.cc:3054] T a3fdab330d574c0283ac58bc9e450b87 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:16.481163 1593 ts_tablet_manager.cc:1905] T 268a12df225e4ba59d4a84c2869d4661 P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:16.490206 1102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bfd8fd6d2104fb4a6484cb8c98981bd" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:16.491269 1102 raft_consensus.cc:2371] T 9bfd8fd6d2104fb4a6484cb8c98981bd P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1. Re-sending same reply.
I20250114 02:54:16.500373 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96b06fe4173343dabd726d253b9f9b5f" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:16.501098 1106 raft_consensus.cc:2463] T 96b06fe4173343dabd726d253b9f9b5f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:16.476181 1506 raft_consensus.cc:513] T 7ecf782bb3d54528aa996abca1ea3a15 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:16.508376 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ecf782bb3d54528aa996abca1ea3a15" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
I20250114 02:54:16.508885 962 raft_consensus.cc:3054] T 7ecf782bb3d54528aa996abca1ea3a15 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:16.508971 1506 leader_election.cc:290] T 7ecf782bb3d54528aa996abca1ea3a15 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Requested vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:16.514109 1597 consensus_queue.cc:1035] T 25e85b0c5c9a4a3ca35a001359f21167 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:16.514390 962 raft_consensus.cc:2463] T 7ecf782bb3d54528aa996abca1ea3a15 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1.
I20250114 02:54:16.515515 1040 leader_election.cc:304] T 7ecf782bb3d54528aa996abca1ea3a15 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:16.516402 1506 raft_consensus.cc:2798] T 7ecf782bb3d54528aa996abca1ea3a15 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:16.516829 1506 raft_consensus.cc:695] T 7ecf782bb3d54528aa996abca1ea3a15 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Becoming Leader. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Running, Role: LEADER
I20250114 02:54:16.517586 1506 consensus_queue.cc:237] T 7ecf782bb3d54528aa996abca1ea3a15 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:16.526389 1206 tablet_bootstrap.cc:654] T 7ecf782bb3d54528aa996abca1ea3a15 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:16.527384 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet b4740f6b3a6741368ba68907ad065279 (table default.Table_8a880e0ae5ff4b5d8de8f40f15dd6e02 [id=6baaeac9e83d4e3092485fca13e07303]) successfully deleted
I20250114 02:54:16.534080 940 tablet_service.cc:1514] Processing DeleteTablet for tablet 9589afd0094545c1801b7a0b5eb8698f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:13 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:16.535204 940 tablet_service.cc:1514] Processing DeleteTablet for tablet b4740f6b3a6741368ba68907ad065279 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:14 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:16.536034 940 tablet_service.cc:1514] Processing DeleteTablet for tablet 83b5b36a26e74668998a19fa0ddf7188 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:12 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:16.533392 898 leader_election.cc:304] T 179248cb5acf4740b268e0ec8b2852f9 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 525198593c8e47afaee54e74267b87e3; no voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:16.543308 1600 raft_consensus.cc:2743] T 179248cb5acf4740b268e0ec8b2852f9 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250114 02:54:16.549886 455 consensus_peers.cc:487] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750 -> Peer 87b3db09ac87436e9a60d6de5fedaff5 (127.0.107.124:39105): Couldn't send request to peer 87b3db09ac87436e9a60d6de5fedaff5. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.124:39105: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20250114 02:54:16.550194 1205 tablet_bootstrap.cc:492] T de4ca83a69704a368a25954cdc1b9392 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:16.550554 1205 ts_tablet_manager.cc:1397] T de4ca83a69704a368a25954cdc1b9392 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.136s user 0.015s sys 0.000s
I20250114 02:54:16.551633 940 tablet_service.cc:1514] Processing DeleteTablet for tablet ed2d137579b8467da5eed4ea2f619a9a with delete_type TABLET_DATA_DELETED (replaced by tablet 0304df55431e42e1a999ed17dc35e6c0) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:16.554534 940 tablet_service.cc:1514] Processing DeleteTablet for tablet c20aca761313498193e1d79c10705f8f with delete_type TABLET_DATA_DELETED (replaced by tablet bc46815b4803418c95ea9c039a4fae88) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:16.553594 1205 raft_consensus.cc:357] T de4ca83a69704a368a25954cdc1b9392 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:16.555150 1205 raft_consensus.cc:383] T de4ca83a69704a368a25954cdc1b9392 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:16.555383 1205 raft_consensus.cc:738] T de4ca83a69704a368a25954cdc1b9392 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:16.556026 1205 consensus_queue.cc:260] T de4ca83a69704a368a25954cdc1b9392 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:16.543414 898 leader_election.cc:304] T 9bfd8fd6d2104fb4a6484cb8c98981bd P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:16.559674 1205 ts_tablet_manager.cc:1428] T de4ca83a69704a368a25954cdc1b9392 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20250114 02:54:16.561664 1600 raft_consensus.cc:2798] T 9bfd8fd6d2104fb4a6484cb8c98981bd P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:16.562111 1600 raft_consensus.cc:695] T 9bfd8fd6d2104fb4a6484cb8c98981bd P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Becoming Leader. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Running, Role: LEADER
I20250114 02:54:16.562855 1600 consensus_queue.cc:237] T 9bfd8fd6d2104fb4a6484cb8c98981bd P 525198593c8e47afaee54e74267b87e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:16.561913 898 leader_election.cc:304] T 96b06fe4173343dabd726d253b9f9b5f P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:16.564908 1205 tablet_bootstrap.cc:492] T 00080bea6c8e46e584ccba6dcb0e221b P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:16.565958 1597 raft_consensus.cc:2798] T 96b06fe4173343dabd726d253b9f9b5f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:16.566341 1597 raft_consensus.cc:491] T 96b06fe4173343dabd726d253b9f9b5f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:16.566751 1597 raft_consensus.cc:3054] T 96b06fe4173343dabd726d253b9f9b5f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:16.567975 1386 raft_consensus.cc:491] T 961b6a25a13a4fa2a732a6383dcd8c84 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:16.568574 1386 raft_consensus.cc:513] T 961b6a25a13a4fa2a732a6383dcd8c84 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:16.570458 1386 leader_election.cc:290] T 961b6a25a13a4fa2a732a6383dcd8c84 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:16.572196 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "961b6a25a13a4fa2a732a6383dcd8c84" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:16.565446 1103 raft_consensus.cc:2463] T a3fdab330d574c0283ac58bc9e450b87 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
W20250114 02:54:16.573626 1040 leader_election.cc:343] T 961b6a25a13a4fa2a732a6383dcd8c84 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:16.575631 815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "961b6a25a13a4fa2a732a6383dcd8c84" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:16.576534 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ecf782bb3d54528aa996abca1ea3a15" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
W20250114 02:54:16.576938 1042 leader_election.cc:343] T 961b6a25a13a4fa2a732a6383dcd8c84 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:16.580191 1042 leader_election.cc:304] T 961b6a25a13a4fa2a732a6383dcd8c84 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90d6c2c58bb54e0eafb365625ac33871; no voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2
W20250114 02:54:16.581303 1042 leader_election.cc:343] T 7ecf782bb3d54528aa996abca1ea3a15 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:16.582151 1505 raft_consensus.cc:2743] T 961b6a25a13a4fa2a732a6383dcd8c84 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:16.576905 1207 tablet_bootstrap.cc:492] T 00080bea6c8e46e584ccba6dcb0e221b P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:16.583266 1207 ts_tablet_manager.cc:1397] T 00080bea6c8e46e584ccba6dcb0e221b P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.491s user 0.011s sys 0.007s
I20250114 02:54:16.597414 1207 raft_consensus.cc:357] T 00080bea6c8e46e584ccba6dcb0e221b P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:16.598126 1207 raft_consensus.cc:383] T 00080bea6c8e46e584ccba6dcb0e221b P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:16.598398 1207 raft_consensus.cc:738] T 00080bea6c8e46e584ccba6dcb0e221b P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:16.614148 1595 ts_tablet_manager.cc:1918] T b4740f6b3a6741368ba68907ad065279 P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:16.614617 1595 log.cc:1198] T b4740f6b3a6741368ba68907ad065279 P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/b4740f6b3a6741368ba68907ad065279
I20250114 02:54:16.615664 1595 ts_tablet_manager.cc:1939] T b4740f6b3a6741368ba68907ad065279 P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
I20250114 02:54:16.622738 1207 consensus_queue.cc:260] T 00080bea6c8e46e584ccba6dcb0e221b P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:16.630621 1618 raft_consensus.cc:491] T de4ca83a69704a368a25954cdc1b9392 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:16.631042 1618 raft_consensus.cc:513] T de4ca83a69704a368a25954cdc1b9392 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
W20250114 02:54:16.632867 456 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 276942e5e54246c79f0d4f08297f58ea (table default.Table_b6330aba5846413c915c50dc4c6e5f11 [id=c3d72f049d9849ab86e252c47e5d24e2], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 276942e5e54246c79f0d4f08297f58ea consensus state has no leader
W20250114 02:54:16.634115 465 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.66:50663 (request call id 39) took 1262 ms (client timeout 1000 ms). Trace:
I20250114 02:54:16.633024 1618 leader_election.cc:290] T de4ca83a69704a368a25954cdc1b9392 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:16.638819 1207 ts_tablet_manager.cc:1428] T 00080bea6c8e46e584ccba6dcb0e221b P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.055s user 0.001s sys 0.004s
I20250114 02:54:16.646039 1594 ts_tablet_manager.cc:1918] T 268a12df225e4ba59d4a84c2869d4661 P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:16.646936 1505 consensus_queue.cc:1035] T e3af8f1299f24b08b2d25811942aaecc P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:16.654795 1597 raft_consensus.cc:513] T 96b06fe4173343dabd726d253b9f9b5f P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:16.671281 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet b4740f6b3a6741368ba68907ad065279 (table default.Table_8a880e0ae5ff4b5d8de8f40f15dd6e02 [id=6baaeac9e83d4e3092485fca13e07303]) successfully deleted
I20250114 02:54:16.672080 1595 tablet_replica.cc:331] T 276942e5e54246c79f0d4f08297f58ea P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:16.672873 1595 raft_consensus.cc:2238] T 276942e5e54246c79f0d4f08297f58ea P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Raft consensus shutting down.
I20250114 02:54:16.673723 1595 raft_consensus.cc:2267] T 276942e5e54246c79f0d4f08297f58ea P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250114 02:54:16.675264 469 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 55b68586ef8b45439a7dd8dcd2efcd79 consensus state has no leader
I20250114 02:54:16.676295 469 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_fa5f4f55ea844a8c8af8acb80f0fd6a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:16.677712 469 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fa5f4f55ea844a8c8af8acb80f0fd6a4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:16.680064 1596 tablet.cc:1721] T b63dd90f60a04465b948171e74888e79 P 90d6c2c58bb54e0eafb365625ac33871: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250114 02:54:16.679811 1594 log.cc:1198] T 268a12df225e4ba59d4a84c2869d4661 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/268a12df225e4ba59d4a84c2869d4661
I20250114 02:54:16.682055 1594 ts_tablet_manager.cc:1939] T 268a12df225e4ba59d4a84c2869d4661 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:16.683847 1638 tablet.cc:1721] T 8b4bc535c42e43889401f14b6d59659e P 90d6c2c58bb54e0eafb365625ac33871: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250114 02:54:16.637214 465 rpcz_store.cc:269] 0114 02:54:15.363821 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:15.441252 (+ 77431us) service_pool.cc:225] Handling call
0114 02:54:15.462868 (+ 21616us) spinlock_profiling.cc:243] Waited 14 ms on lock 0x7b54000316d4. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616cad50 00007f646174df2a 00007f64633f2dec 00007f64634eadee 00007f6461248fc4 00007f6461248f52 00007f6461248e81 00007f6461248dfc 00007f64612480b2 00007f645f512dfc 00007f645f512236
0114 02:54:16.626368 (+1163500us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:15.919514 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:15.919988 (+ 474us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:15.920024 (+ 36us) write_op.cc:435] Acquired schema lock
0114 02:54:15.920043 (+ 19us) tablet.cc:659] Decoding operations
0114 02:54:15.920407 (+ 364us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:15.920458 (+ 51us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:15.920491 (+ 33us) tablet.cc:682] Acquiring locks for 1 operations
0114 02:54:15.920673 (+ 182us) tablet.cc:698] Row locks acquired
0114 02:54:15.920692 (+ 19us) write_op.cc:260] PREPARE: finished
0114 02:54:15.920822 (+ 130us) write_op.cc:270] Start()
0114 02:54:15.920924 (+ 102us) write_op.cc:276] Timestamp: P: 1736823255920670 usec, L: 0
0114 02:54:15.920945 (+ 21us) op_driver.cc:348] REPLICATION: starting
0114 02:54:16.009800 (+ 88855us) log.cc:844] Serialized 530 byte log entry
0114 02:54:16.012192 (+ 2392us) spinlock_profiling.cc:243] Waited 81.5 ms on lock 0x7b54000316d4. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616ca6d1 00007f6461741198 00007f6461bad3b1 00007f6461bacb88 00007f6461bb1821 00007f6461bb17d9 00007f6461bb1769 00007f6461bb1731 00007f6461bb095d 000000000059ea04 000000000059e839
0114 02:54:16.044855 (+ 32663us) op_driver.cc:464] REPLICATION: finished
0114 02:54:16.045246 (+ 391us) write_op.cc:301] APPLY: starting
0114 02:54:16.045325 (+ 79us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:16.045508 (+ 183us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:16.045535 (+ 27us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:16.045735 (+ 200us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:16.045903 (+ 168us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:16.045966 (+ 63us) write_op.cc:312] APPLY: finished
0114 02:54:16.046169 (+ 203us) log.cc:844] Serialized 23 byte log entry
0114 02:54:16.046861 (+ 692us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:16.046921 (+ 60us) write_op.cc:454] Released schema lock
0114 02:54:16.046953 (+ 32us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:16.047187 (+ 234us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:16.047221 (+ 34us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":14,"spinlock_wait_cycles":32296704,"child_traces":[["op",{"apply.queue_time_us":180,"apply.run_cpu_time_us":1338,"apply.run_wall_time_us":1912,"num_ops":1,"prepare.queue_time_us":111,"prepare.run_cpu_time_us":5039,"prepare.run_wall_time_us":92979,"raft.queue_time_us":94,"raft.run_cpu_time_us":644,"raft.run_wall_time_us":644,"replication_time_us":123167,"spinlock_wait_cycles":187693184}]]}
I20250114 02:54:16.685953 1205 tablet_bootstrap.cc:654] T 00080bea6c8e46e584ccba6dcb0e221b P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:16.686365 465 master_service.cc:946] ReplaceTablet: received request to replace tablet b63dd90f60a04465b948171e74888e79 from {username='slave'} at 127.0.0.1:33540
I20250114 02:54:16.694574 1099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de4ca83a69704a368a25954cdc1b9392" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:16.695228 1099 raft_consensus.cc:2463] T de4ca83a69704a368a25954cdc1b9392 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:16.697041 1207 tablet_bootstrap.cc:492] T db24bf3db61941ba9b5e0662db23e748 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:16.698554 816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de4ca83a69704a368a25954cdc1b9392" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:16.699666 1597 leader_election.cc:290] T 96b06fe4173343dabd726d253b9f9b5f P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Requested vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:16.701350 1099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96b06fe4173343dabd726d253b9f9b5f" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:16.701929 1099 raft_consensus.cc:3054] T 96b06fe4173343dabd726d253b9f9b5f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:16.677807 1206 tablet_bootstrap.cc:492] T 7ecf782bb3d54528aa996abca1ea3a15 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:16.704399 1206 ts_tablet_manager.cc:1397] T 7ecf782bb3d54528aa996abca1ea3a15 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.549s user 0.020s sys 0.019s
I20250114 02:54:16.706475 1206 raft_consensus.cc:357] T 7ecf782bb3d54528aa996abca1ea3a15 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:16.710368 816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96b06fe4173343dabd726d253b9f9b5f" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:16.710894 816 raft_consensus.cc:3054] T 96b06fe4173343dabd726d253b9f9b5f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:16.717815 1593 ts_tablet_manager.cc:1918] T 268a12df225e4ba59d4a84c2869d4661 P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:16.725344 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet 268a12df225e4ba59d4a84c2869d4661 (table default.Table_a1731c2cc5964c3aaddb43ec22096aad [id=67870401767c48be838c7163a9ba8beb]) successfully deleted
I20250114 02:54:16.725636 1594 tablet_replica.cc:331] T 83b5b36a26e74668998a19fa0ddf7188 P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:16.726550 1594 raft_consensus.cc:2238] T 83b5b36a26e74668998a19fa0ddf7188 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:16.727128 1594 raft_consensus.cc:2267] T 83b5b36a26e74668998a19fa0ddf7188 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:16.734544 1506 consensus_queue.cc:1035] T e3af8f1299f24b08b2d25811942aaecc P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:16.707074 1206 raft_consensus.cc:383] T 7ecf782bb3d54528aa996abca1ea3a15 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:16.736591 1206 raft_consensus.cc:738] T 7ecf782bb3d54528aa996abca1ea3a15 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:16.739180 1207 tablet_bootstrap.cc:654] T db24bf3db61941ba9b5e0662db23e748 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:16.741216 1206 consensus_queue.cc:260] T 7ecf782bb3d54528aa996abca1ea3a15 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:16.750792 464 master_service.cc:946] ReplaceTablet: received request to replace tablet 961b6a25a13a4fa2a732a6383dcd8c84 from {username='slave'} at 127.0.0.1:33540
I20250114 02:54:16.753968 791 tablet_service.cc:1467] Processing CreateTablet for tablet 71bd00f7d8ac41839f36fd73997631ad (DEFAULT_TABLE table=default.Table_3f4f2aa33eae44a1ba3c243179bfd590 [id=b46c2d8e8a2b4c3b8d55c78e6c0dc401]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:16.754295 1594 ts_tablet_manager.cc:1905] T 83b5b36a26e74668998a19fa0ddf7188 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:16.755077 791 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71bd00f7d8ac41839f36fd73997631ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:16.758605 790 tablet_service.cc:1467] Processing CreateTablet for tablet 044832851f804208b0a371eb0857dbf3 (DEFAULT_TABLE table=default.Table_9a7ce95817ea4e4fb4bc3d8ccb6f5e8c [id=515a2fe3f1e84e82bccc02ef5cecef1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:16.759617 790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 044832851f804208b0a371eb0857dbf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:16.762322 1206 ts_tablet_manager.cc:1428] T 7ecf782bb3d54528aa996abca1ea3a15 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.058s user 0.005s sys 0.001s
I20250114 02:54:16.763124 1206 tablet_bootstrap.cc:492] T 961b6a25a13a4fa2a732a6383dcd8c84 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:16.750303 940 tablet_service.cc:1467] Processing CreateTablet for tablet 71bd00f7d8ac41839f36fd73997631ad (DEFAULT_TABLE table=default.Table_3f4f2aa33eae44a1ba3c243179bfd590 [id=b46c2d8e8a2b4c3b8d55c78e6c0dc401]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:16.772236 940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71bd00f7d8ac41839f36fd73997631ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:16.751295 936 tablet_service.cc:1467] Processing CreateTablet for tablet 044832851f804208b0a371eb0857dbf3 (DEFAULT_TABLE table=default.Table_9a7ce95817ea4e4fb4bc3d8ccb6f5e8c [id=515a2fe3f1e84e82bccc02ef5cecef1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:16.776803 936 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 044832851f804208b0a371eb0857dbf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:16.786245 939 tablet_service.cc:1467] Processing CreateTablet for tablet f2b6812e8c1d44529d0284be490df648 (DEFAULT_TABLE table=default.Table_b9119494def445ddba465df3e4513efe [id=eb4927e72691401a979d7e91fe149458]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:16.786420 941 tablet_service.cc:1467] Processing CreateTablet for tablet d3fb28717c6f4844af84bb817fe7dfc4 (DEFAULT_TABLE table=default.Table_b8b1e6d9bd0742f7880a7f55e986d0ee [id=871e09c142b94b79aeff9bfd8d78e2bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:16.787304 939 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2b6812e8c1d44529d0284be490df648. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:16.787513 937 tablet_service.cc:1467] Processing CreateTablet for tablet ca986f81ed6848a89588366f3f400eec (DEFAULT_TABLE table=default.Table_123095518d164d3db7fe3bd50af9dbd4 [id=4744eda447ca4785a9c065aca94135e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:16.788727 937 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca986f81ed6848a89588366f3f400eec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:16.793654 469 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_debb45fd7c4b4826a73d39a6a6df88c4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:16.795049 469 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_debb45fd7c4b4826a73d39a6a6df88c4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:16.795336 798 tablet_service.cc:1467] Processing CreateTablet for tablet ca986f81ed6848a89588366f3f400eec (DEFAULT_TABLE table=default.Table_123095518d164d3db7fe3bd50af9dbd4 [id=4744eda447ca4785a9c065aca94135e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:16.796294 798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca986f81ed6848a89588366f3f400eec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:16.796895 794 tablet_service.cc:1467] Processing CreateTablet for tablet f2b6812e8c1d44529d0284be490df648 (DEFAULT_TABLE table=default.Table_b9119494def445ddba465df3e4513efe [id=eb4927e72691401a979d7e91fe149458]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:16.797917 794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2b6812e8c1d44529d0284be490df648. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:16.801254 795 tablet_service.cc:1467] Processing CreateTablet for tablet 0304df55431e42e1a999ed17dc35e6c0 (DEFAULT_TABLE table=default.Table_4c3843233dda407395ab9652d5870bc6 [id=0d173fdb502f482fb86b6d3e5fdc66f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:16.801295 792 tablet_service.cc:1467] Processing CreateTablet for tablet d3fb28717c6f4844af84bb817fe7dfc4 (DEFAULT_TABLE table=default.Table_b8b1e6d9bd0742f7880a7f55e986d0ee [id=871e09c142b94b79aeff9bfd8d78e2bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:16.802299 795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0304df55431e42e1a999ed17dc35e6c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:16.802543 1593 log.cc:1198] T 268a12df225e4ba59d4a84c2869d4661 P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/268a12df225e4ba59d4a84c2869d4661
I20250114 02:54:16.802731 941 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3fb28717c6f4844af84bb817fe7dfc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:16.803297 1593 ts_tablet_manager.cc:1939] T 268a12df225e4ba59d4a84c2869d4661 P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:16.806149 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet 268a12df225e4ba59d4a84c2869d4661 (table default.Table_a1731c2cc5964c3aaddb43ec22096aad [id=67870401767c48be838c7163a9ba8beb]) successfully deleted
I20250114 02:54:16.807415 1593 tablet_replica.cc:331] T 276942e5e54246c79f0d4f08297f58ea P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:16.807533 792 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3fb28717c6f4844af84bb817fe7dfc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:16.768723 1085 tablet_service.cc:1467] Processing CreateTablet for tablet 71bd00f7d8ac41839f36fd73997631ad (DEFAULT_TABLE table=default.Table_3f4f2aa33eae44a1ba3c243179bfd590 [id=b46c2d8e8a2b4c3b8d55c78e6c0dc401]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:16.829622 1206 tablet_bootstrap.cc:654] T 961b6a25a13a4fa2a732a6383dcd8c84 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:16.844769 1594 ts_tablet_manager.cc:1918] T 83b5b36a26e74668998a19fa0ddf7188 P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:16.845191 1594 log.cc:1198] T 83b5b36a26e74668998a19fa0ddf7188 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/83b5b36a26e74668998a19fa0ddf7188
W20250114 02:54:16.847193 455 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.125:46837, user_credentials={real_user=slave}} blocked reactor thread for 57743.9us
I20250114 02:54:16.860991 1086 tablet_service.cc:1467] Processing CreateTablet for tablet d3fb28717c6f4844af84bb817fe7dfc4 (DEFAULT_TABLE table=default.Table_b8b1e6d9bd0742f7880a7f55e986d0ee [id=871e09c142b94b79aeff9bfd8d78e2bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:16.850123 1595 ts_tablet_manager.cc:1905] T 276942e5e54246c79f0d4f08297f58ea P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:16.862246 1080 tablet_service.cc:1467] Processing CreateTablet for tablet f2b6812e8c1d44529d0284be490df648 (DEFAULT_TABLE table=default.Table_b9119494def445ddba465df3e4513efe [id=eb4927e72691401a979d7e91fe149458]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:16.860991 1082 tablet_service.cc:1467] Processing CreateTablet for tablet ca986f81ed6848a89588366f3f400eec (DEFAULT_TABLE table=default.Table_123095518d164d3db7fe3bd50af9dbd4 [id=4744eda447ca4785a9c065aca94135e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:16.874209 1081 tablet_service.cc:1467] Processing CreateTablet for tablet 0304df55431e42e1a999ed17dc35e6c0 (DEFAULT_TABLE table=default.Table_4c3843233dda407395ab9652d5870bc6 [id=0d173fdb502f482fb86b6d3e5fdc66f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:16.857867 1083 tablet_service.cc:1467] Processing CreateTablet for tablet 044832851f804208b0a371eb0857dbf3 (DEFAULT_TABLE table=default.Table_9a7ce95817ea4e4fb4bc3d8ccb6f5e8c [id=515a2fe3f1e84e82bccc02ef5cecef1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:16.888432 467 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet 179248cb5acf4740b268e0ec8b2852f9 (table default.Table_bd405598fe054b718a8d6fb5dd875f0e [id=5745501ecd7b4cecaca3d35502a6cc40]). Expected version 1 got 0
I20250114 02:54:16.888990 467 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet 8b4bc535c42e43889401f14b6d59659e (table default.Table_d90b2995b13b4a8fb7e50cc4b79cbf84 [id=f8e40e0aa44b4157ac3438d11685b2e7]). Expected version 1 got 0
I20250114 02:54:16.890087 467 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet a3fdab330d574c0283ac58bc9e450b87 (table default.Table_ec0eefb64458440ebd57a05a7ad0bb9a [id=b6874c22fe0d4151a7c8ea061f685938]). Expected version 1 got 0
I20250114 02:54:16.892935 465 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet b63dd90f60a04465b948171e74888e79
I20250114 02:54:16.894217 465 catalog_manager.cc:6596] ReplaceTablet: tablet b63dd90f60a04465b948171e74888e79 deleted and replaced by tablet 59a46bc865cc48df8e98598764fd1b54
I20250114 02:54:16.836936 1618 raft_consensus.cc:491] T 15704785f56e43a3af69f181c75877ce P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:16.865118 938 tablet_service.cc:1467] Processing CreateTablet for tablet 0304df55431e42e1a999ed17dc35e6c0 (DEFAULT_TABLE table=default.Table_4c3843233dda407395ab9652d5870bc6 [id=0d173fdb502f482fb86b6d3e5fdc66f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:16.897112 1618 raft_consensus.cc:513] T 15704785f56e43a3af69f181c75877ce P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:16.896777 467 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet 4bb9b429af92446bbe13894d5dfe4c47 (table default.Table_acb5bf9354be440bbf6034cf1c535dab [id=22b7edcf4b7e4c31b3c23dc44ccf028e]). Expected version 1 got 0
I20250114 02:54:16.898931 898 leader_election.cc:304] T de4ca83a69704a368a25954cdc1b9392 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:16.899597 1601 raft_consensus.cc:2798] T de4ca83a69704a368a25954cdc1b9392 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:16.899932 1601 raft_consensus.cc:491] T de4ca83a69704a368a25954cdc1b9392 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:16.900151 1618 leader_election.cc:290] T 15704785f56e43a3af69f181c75877ce P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:16.900246 1601 raft_consensus.cc:3054] T de4ca83a69704a368a25954cdc1b9392 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:16.896579 940 tablet_service.cc:1514] Processing DeleteTablet for tablet b63dd90f60a04465b948171e74888e79 with delete_type TABLET_DATA_DELETED (replaced by tablet 59a46bc865cc48df8e98598764fd1b54) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:16.908520 938 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0304df55431e42e1a999ed17dc35e6c0. 1 dirs total, 0 dirs full, 0 dirs failed
W20250114 02:54:16.909430 898 leader_election.cc:343] T de4ca83a69704a368a25954cdc1b9392 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:16.911154 815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15704785f56e43a3af69f181c75877ce" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:16.907554 1601 raft_consensus.cc:513] T de4ca83a69704a368a25954cdc1b9392 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:16.913475 1593 raft_consensus.cc:2238] T 276942e5e54246c79f0d4f08297f58ea P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:16.914106 1601 leader_election.cc:290] T de4ca83a69704a368a25954cdc1b9392 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Requested vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:16.914196 1593 raft_consensus.cc:2267] T 276942e5e54246c79f0d4f08297f58ea P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:16.915277 462 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 4bb9b429af92446bbe13894d5dfe4c47 (table default.Table_acb5bf9354be440bbf6034cf1c535dab [id=22b7edcf4b7e4c31b3c23dc44ccf028e]). Expected version 1 got 0
I20250114 02:54:16.915757 1594 ts_tablet_manager.cc:1939] T 83b5b36a26e74668998a19fa0ddf7188 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:16.917754 1594 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 101) took 1077 ms. Trace:
I20250114 02:54:16.920063 462 catalog_manager.cc:5526] T 179248cb5acf4740b268e0ec8b2852f9 P 819f9e3412e54d44abb11d8b3ca877c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65). New cstate: current_term: 1 leader_uuid: "819f9e3412e54d44abb11d8b3ca877c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:16.926518 1085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71bd00f7d8ac41839f36fd73997631ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:16.929427 1086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3fb28717c6f4844af84bb817fe7dfc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:16.934454 462 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 179248cb5acf4740b268e0ec8b2852f9 (table default.Table_bd405598fe054b718a8d6fb5dd875f0e [id=5745501ecd7b4cecaca3d35502a6cc40]). Expected version 1 got 0
I20250114 02:54:16.936480 1080 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2b6812e8c1d44529d0284be490df648. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:16.940881 1082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca986f81ed6848a89588366f3f400eec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:16.947094 1081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0304df55431e42e1a999ed17dc35e6c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:16.955621 1206 tablet_bootstrap.cc:492] T 961b6a25a13a4fa2a732a6383dcd8c84 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:16.956074 1206 ts_tablet_manager.cc:1397] T 961b6a25a13a4fa2a732a6383dcd8c84 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.193s user 0.008s sys 0.002s
I20250114 02:54:16.958245 1206 raft_consensus.cc:357] T 961b6a25a13a4fa2a732a6383dcd8c84 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:16.958815 1206 raft_consensus.cc:383] T 961b6a25a13a4fa2a732a6383dcd8c84 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:16.959069 1206 raft_consensus.cc:738] T 961b6a25a13a4fa2a732a6383dcd8c84 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:16.959669 1206 consensus_queue.cc:260] T 961b6a25a13a4fa2a732a6383dcd8c84 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:16.960319 1593 ts_tablet_manager.cc:1905] T 276942e5e54246c79f0d4f08297f58ea P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:16.960855 1083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 044832851f804208b0a371eb0857dbf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:16.961187 1206 ts_tablet_manager.cc:1428] T 961b6a25a13a4fa2a732a6383dcd8c84 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250114 02:54:16.961968 1206 tablet_bootstrap.cc:492] T 00080bea6c8e46e584ccba6dcb0e221b P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:16.965538 1507 raft_consensus.cc:491] T 00080bea6c8e46e584ccba6dcb0e221b P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:16.965981 1507 raft_consensus.cc:513] T 00080bea6c8e46e584ccba6dcb0e221b P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:16.967765 1507 leader_election.cc:290] T 00080bea6c8e46e584ccba6dcb0e221b P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:16.969790 1206 tablet_bootstrap.cc:654] T 00080bea6c8e46e584ccba6dcb0e221b P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:16.970677 1084 tablet_service.cc:1514] Processing DeleteTablet for tablet b63dd90f60a04465b948171e74888e79 with delete_type TABLET_DATA_DELETED (replaced by tablet 59a46bc865cc48df8e98598764fd1b54) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:16.973661 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00080bea6c8e46e584ccba6dcb0e221b" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:16.980921 1206 tablet_bootstrap.cc:492] T 00080bea6c8e46e584ccba6dcb0e221b P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:16.981300 1206 ts_tablet_manager.cc:1397] T 00080bea6c8e46e584ccba6dcb0e221b P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.003s
I20250114 02:54:16.981367 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00080bea6c8e46e584ccba6dcb0e221b" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:16.983916 1206 raft_consensus.cc:357] T 00080bea6c8e46e584ccba6dcb0e221b P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:16.984568 1206 raft_consensus.cc:383] T 00080bea6c8e46e584ccba6dcb0e221b P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:16.984805 1206 raft_consensus.cc:738] T 00080bea6c8e46e584ccba6dcb0e221b P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:16.987130 1206 consensus_queue.cc:260] T 00080bea6c8e46e584ccba6dcb0e221b P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:16.987962 815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de4ca83a69704a368a25954cdc1b9392" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
W20250114 02:54:16.986496 1040 leader_election.cc:343] T 00080bea6c8e46e584ccba6dcb0e221b P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:16.944648 465 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_77652aa0d7a94f1f98afb881a924eb24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:16.991061 462 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 8b4bc535c42e43889401f14b6d59659e (table default.Table_d90b2995b13b4a8fb7e50cc4b79cbf84 [id=f8e40e0aa44b4157ac3438d11685b2e7]). Expected version 1 got 0
I20250114 02:54:16.991382 818 raft_consensus.cc:2463] T 00080bea6c8e46e584ccba6dcb0e221b P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 0.
W20250114 02:54:16.991591 465 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_77652aa0d7a94f1f98afb881a924eb24 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:16.992086 462 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet a3fdab330d574c0283ac58bc9e450b87 (table default.Table_ec0eefb64458440ebd57a05a7ad0bb9a [id=b6874c22fe0d4151a7c8ea061f685938]). Expected version 1 got 0
I20250114 02:54:16.993204 1206 ts_tablet_manager.cc:1428] T 00080bea6c8e46e584ccba6dcb0e221b P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20250114 02:54:16.995762 1206 tablet_bootstrap.cc:492] T 9812754c610548d1b43084bbc94fc3e4 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:17.001292 1206 tablet_bootstrap.cc:654] T 9812754c610548d1b43084bbc94fc3e4 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
W20250114 02:54:17.002635 898 leader_election.cc:343] T 15704785f56e43a3af69f181c75877ce P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Illegal state: must be running to vote when last-logged opid is not known
W20250114 02:54:17.003347 898 leader_election.cc:343] T de4ca83a69704a368a25954cdc1b9392 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:17.007944 1101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15704785f56e43a3af69f181c75877ce" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:17.008575 1101 raft_consensus.cc:2463] T 15704785f56e43a3af69f181c75877ce P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:17.009455 1593 ts_tablet_manager.cc:1918] T 276942e5e54246c79f0d4f08297f58ea P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:17.009816 1593 log.cc:1198] T 276942e5e54246c79f0d4f08297f58ea P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/276942e5e54246c79f0d4f08297f58ea
I20250114 02:54:17.010517 1593 ts_tablet_manager.cc:1939] T 276942e5e54246c79f0d4f08297f58ea P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:17.000116 1042 leader_election.cc:304] T 00080bea6c8e46e584ccba6dcb0e221b P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters: 525198593c8e47afaee54e74267b87e3
I20250114 02:54:17.018615 1505 raft_consensus.cc:2798] T 00080bea6c8e46e584ccba6dcb0e221b P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:17.018944 1505 raft_consensus.cc:491] T 00080bea6c8e46e584ccba6dcb0e221b P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:17.019238 1505 raft_consensus.cc:3054] T 00080bea6c8e46e584ccba6dcb0e221b P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:17.027612 816 raft_consensus.cc:2463] T 96b06fe4173343dabd726d253b9f9b5f P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:17.028733 1593 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55068 (request call id 107) took 1159 ms. Trace:
I20250114 02:54:17.029263 1593 rpcz_store.cc:276] 0114 02:54:15.868865 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:15.868985 (+ 120us) service_pool.cc:225] Handling call
0114 02:54:16.807258 (+938273us) ts_tablet_manager.cc:1145] Deleting tablet 276942e5e54246c79f0d4f08297f58ea
0114 02:54:16.807307 (+ 49us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:16.967848 (+160541us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:17.009225 (+ 41377us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:17.028638 (+ 19413us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":32933,"tablet-delete.queue_time_us":937745}
I20250114 02:54:17.032522 1593 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55068 (request call id 108) took 1110 ms. Trace:
I20250114 02:54:17.032820 1593 rpcz_store.cc:276] 0114 02:54:15.922058 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:15.922192 (+ 134us) service_pool.cc:225] Handling call
0114 02:54:17.032143 (+1109951us) ts_tablet_manager.cc:1145] Deleting tablet db24bf3db61941ba9b5e0662db23e748
0114 02:54:17.032189 (+ 46us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:17.032472 (+ 283us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1109207}
I20250114 02:54:17.033799 1593 tablet_replica.cc:331] T 96b06fe4173343dabd726d253b9f9b5f P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:17.035125 1593 raft_consensus.cc:2238] T 96b06fe4173343dabd726d253b9f9b5f P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:17.035614 1593 raft_consensus.cc:2267] T 96b06fe4173343dabd726d253b9f9b5f P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:17.037454 1593 ts_tablet_manager.cc:1905] T 96b06fe4173343dabd726d253b9f9b5f P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:17.037379 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet 276942e5e54246c79f0d4f08297f58ea (table default.Table_b6330aba5846413c915c50dc4c6e5f11 [id=c3d72f049d9849ab86e252c47e5d24e2]) successfully deleted
W20250114 02:54:17.038010 456 catalog_manager.cc:4852] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet db24bf3db61941ba9b5e0662db23e748 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet db24bf3db61941ba9b5e0662db23e748 already in progress: creating tablet
I20250114 02:54:17.040272 1101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de4ca83a69704a368a25954cdc1b9392" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:17.040802 1101 raft_consensus.cc:3054] T de4ca83a69704a368a25954cdc1b9392 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:17.044441 1207 tablet_bootstrap.cc:492] T db24bf3db61941ba9b5e0662db23e748 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:17.045003 1207 ts_tablet_manager.cc:1397] T db24bf3db61941ba9b5e0662db23e748 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.369s user 0.012s sys 0.007s
I20250114 02:54:16.985436 1099 raft_consensus.cc:2463] T 96b06fe4173343dabd726d253b9f9b5f P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:17.051134 1593 ts_tablet_manager.cc:1918] T 96b06fe4173343dabd726d253b9f9b5f P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:17.051517 1593 log.cc:1198] T 96b06fe4173343dabd726d253b9f9b5f P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/96b06fe4173343dabd726d253b9f9b5f
I20250114 02:54:17.052163 1593 ts_tablet_manager.cc:1939] T 96b06fe4173343dabd726d253b9f9b5f P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:17.053211 1593 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55068 (request call id 111) took 1018 ms. Trace:
I20250114 02:54:17.053498 1593 rpcz_store.cc:276] 0114 02:54:16.034799 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:16.034975 (+ 176us) service_pool.cc:225] Handling call
0114 02:54:17.033659 (+998684us) ts_tablet_manager.cc:1145] Deleting tablet 96b06fe4173343dabd726d253b9f9b5f
0114 02:54:17.033697 (+ 38us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:17.044506 (+ 10809us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:17.050913 (+ 6407us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:17.053158 (+ 2245us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":998156}
I20250114 02:54:17.054589 1593 tablet_replica.cc:331] T 9589afd0094545c1801b7a0b5eb8698f P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:17.054566 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet 96b06fe4173343dabd726d253b9f9b5f (table default.Table_d1de82caa5b94791b6c7dfed1c376e8a [id=03c775a54e9e443cbc4fab853bd6d49c]) successfully deleted
I20250114 02:54:17.055287 1593 raft_consensus.cc:2238] T 9589afd0094545c1801b7a0b5eb8698f P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:17.033769 1206 tablet_bootstrap.cc:492] T 9812754c610548d1b43084bbc94fc3e4 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:17.055876 1206 ts_tablet_manager.cc:1397] T 9812754c610548d1b43084bbc94fc3e4 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.062s user 0.014s sys 0.007s
I20250114 02:54:16.918126 1594 rpcz_store.cc:276] 0114 02:54:15.840326 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:15.840425 (+ 99us) service_pool.cc:225] Handling call
0114 02:54:16.725480 (+885055us) ts_tablet_manager.cc:1145] Deleting tablet 83b5b36a26e74668998a19fa0ddf7188
0114 02:54:16.725523 (+ 43us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:16.764210 (+ 38687us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:16.844521 (+ 80311us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:16.917697 (+ 73176us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":63,"tablet-delete.queue_time_us":884522}
I20250114 02:54:17.055728 1593 raft_consensus.cc:2267] T 9589afd0094545c1801b7a0b5eb8698f P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:17.056699 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet 83b5b36a26e74668998a19fa0ddf7188 (table default.Table_acca64d3328d47578d6d1ebd594babbd [id=bb08f9cb85c44d108156675ff51b37c1]) successfully deleted
I20250114 02:54:17.057255 898 leader_election.cc:304] T 96b06fe4173343dabd726d253b9f9b5f P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
W20250114 02:54:17.057864 898 raft_consensus.cc:2681] T 96b06fe4173343dabd726d253b9f9b5f P 525198593c8e47afaee54e74267b87e3: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250114 02:54:17.057996 1206 raft_consensus.cc:357] T 9812754c610548d1b43084bbc94fc3e4 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:17.058568 1206 raft_consensus.cc:383] T 9812754c610548d1b43084bbc94fc3e4 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:17.058787 1206 raft_consensus.cc:738] T 9812754c610548d1b43084bbc94fc3e4 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:17.059274 1206 consensus_queue.cc:260] T 9812754c610548d1b43084bbc94fc3e4 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:17.061106 1206 ts_tablet_manager.cc:1428] T 9812754c610548d1b43084bbc94fc3e4 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250114 02:54:17.054071 795 tablet_service.cc:1514] Processing DeleteTablet for tablet b63dd90f60a04465b948171e74888e79 with delete_type TABLET_DATA_DELETED (replaced by tablet 59a46bc865cc48df8e98598764fd1b54) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:17.072598 898 leader_election.cc:304] T 15704785f56e43a3af69f181c75877ce P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters: 819f9e3412e54d44abb11d8b3ca877c2
I20250114 02:54:17.074215 1597 raft_consensus.cc:2798] T 15704785f56e43a3af69f181c75877ce P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:17.074533 1597 raft_consensus.cc:491] T 15704785f56e43a3af69f181c75877ce P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:17.074826 1597 raft_consensus.cc:3054] T 15704785f56e43a3af69f181c75877ce P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:17.077582 1206 tablet_bootstrap.cc:492] T db24bf3db61941ba9b5e0662db23e748 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:17.081449 1597 raft_consensus.cc:513] T 15704785f56e43a3af69f181c75877ce P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:17.083310 1597 leader_election.cc:290] T 15704785f56e43a3af69f181c75877ce P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Requested vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:17.084931 816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15704785f56e43a3af69f181c75877ce" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:17.085441 1594 tablet_replica.cc:331] T 9589afd0094545c1801b7a0b5eb8698f P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:17.086812 1594 raft_consensus.cc:2238] T 9589afd0094545c1801b7a0b5eb8698f P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 LEADER]: Raft consensus shutting down.
I20250114 02:54:17.085224 1206 tablet_bootstrap.cc:654] T db24bf3db61941ba9b5e0662db23e748 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
W20250114 02:54:17.089864 898 leader_election.cc:343] T 15704785f56e43a3af69f181c75877ce P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:17.058566 1205 tablet_bootstrap.cc:492] T 00080bea6c8e46e584ccba6dcb0e221b P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:17.063159 1593 ts_tablet_manager.cc:1905] T 9589afd0094545c1801b7a0b5eb8698f P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:17.094300 1205 ts_tablet_manager.cc:1397] T 00080bea6c8e46e584ccba6dcb0e221b P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.530s user 0.011s sys 0.000s
I20250114 02:54:17.098435 463 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet 99b8acb1fe594bf3825c7b9121018db6
I20250114 02:54:17.101783 937 tablet_service.cc:1514] Processing DeleteTablet for tablet 99b8acb1fe594bf3825c7b9121018db6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:16 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:17.103305 1084 tablet_service.cc:1514] Processing DeleteTablet for tablet 99b8acb1fe594bf3825c7b9121018db6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:16 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:17.100116 463 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:33540 (request call id 166) took 1025 ms. Trace:
I20250114 02:54:17.104106 1207 raft_consensus.cc:357] T db24bf3db61941ba9b5e0662db23e748 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:17.104924 1207 raft_consensus.cc:383] T db24bf3db61941ba9b5e0662db23e748 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:17.105186 1207 raft_consensus.cc:738] T db24bf3db61941ba9b5e0662db23e748 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:17.105762 1207 consensus_queue.cc:260] T db24bf3db61941ba9b5e0662db23e748 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:17.103971 463 rpcz_store.cc:276] 0114 02:54:16.075130 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:16.282790 (+207660us) service_pool.cc:225] Handling call
0114 02:54:16.284134 (+ 1344us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0114 02:54:16.285615 (+ 1481us) catalog_manager.cc:2753] Modifying in-memory table state
0114 02:54:16.285833 (+ 218us) catalog_manager.cc:2762] Locking tablets
0114 02:54:16.757268 (+471435us) catalog_manager.cc:2776] Removing table and tablets from system table
0114 02:54:17.097997 (+340729us) catalog_manager.cc:2794] Removing table from by-name map
0114 02:54:17.098227 (+ 230us) catalog_manager.cc:2810] Committing in-memory state
0114 02:54:17.098339 (+ 112us) catalog_manager.cc:2814] Aborting table tasks
0114 02:54:17.100055 (+ 1716us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:17.006857 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:17.007330 (+ 473us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:17.007361 (+ 31us) write_op.cc:435] Acquired schema lock
0114 02:54:17.007380 (+ 19us) tablet.cc:659] Decoding operations
0114 02:54:17.054555 (+ 47175us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:17.054600 (+ 45us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:17.054621 (+ 21us) tablet.cc:682] Acquiring locks for 2 operations
0114 02:54:17.055891 (+ 1270us) tablet.cc:698] Row locks acquired
0114 02:54:17.055906 (+ 15us) write_op.cc:260] PREPARE: finished
0114 02:54:17.056035 (+ 129us) write_op.cc:270] Start()
0114 02:54:17.073144 (+ 17109us) write_op.cc:276] Timestamp: P: 1736823257055933 usec, L: 0
0114 02:54:17.073166 (+ 22us) op_driver.cc:348] REPLICATION: starting
0114 02:54:17.073642 (+ 476us) log.cc:844] Serialized 728 byte log entry
0114 02:54:17.095988 (+ 22346us) op_driver.cc:464] REPLICATION: finished
0114 02:54:17.096453 (+ 465us) write_op.cc:301] APPLY: starting
0114 02:54:17.096506 (+ 53us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:17.096619 (+ 113us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:17.096707 (+ 88us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:17.096953 (+ 246us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:17.097080 (+ 127us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0114 02:54:17.097119 (+ 39us) write_op.cc:312] APPLY: finished
0114 02:54:17.097297 (+ 178us) log.cc:844] Serialized 29 byte log entry
0114 02:54:17.097408 (+ 111us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:17.097468 (+ 60us) write_op.cc:454] Released schema lock
0114 02:54:17.097495 (+ 27us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:17.097711 (+ 216us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:17.097739 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":309,"apply.run_cpu_time_us":1228,"apply.run_wall_time_us":1228,"num_ops":2,"prepare.queue_time_us":247827,"prepare.run_cpu_time_us":2411,"prepare.run_wall_time_us":79846,"raft.queue_time_us":11977,"raft.run_cpu_time_us":633,"raft.run_wall_time_us":633,"replication_time_us":22739}]]}
I20250114 02:54:17.107904 1207 ts_tablet_manager.cc:1428] T db24bf3db61941ba9b5e0662db23e748 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.063s user 0.003s sys 0.000s
I20250114 02:54:17.108736 1505 raft_consensus.cc:513] T 00080bea6c8e46e584ccba6dcb0e221b P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:17.109549 1593 ts_tablet_manager.cc:1918] T 9589afd0094545c1801b7a0b5eb8698f P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:17.109927 1593 log.cc:1198] T 9589afd0094545c1801b7a0b5eb8698f P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/9589afd0094545c1801b7a0b5eb8698f
I20250114 02:54:17.110538 1505 leader_election.cc:290] T 00080bea6c8e46e584ccba6dcb0e221b P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Requested vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:17.110636 1593 ts_tablet_manager.cc:1939] T 9589afd0094545c1801b7a0b5eb8698f P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:17.112990 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00080bea6c8e46e584ccba6dcb0e221b" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
I20250114 02:54:17.113862 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet 9589afd0094545c1801b7a0b5eb8698f (table default.Table_ed523e66ce6945a7a0fcdb96e498131d [id=aad0de2f82d744ce9054dc12a6b55536]) successfully deleted
I20250114 02:54:17.114271 1595 ts_tablet_manager.cc:1918] T 276942e5e54246c79f0d4f08297f58ea P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:17.114642 1595 log.cc:1198] T 276942e5e54246c79f0d4f08297f58ea P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/276942e5e54246c79f0d4f08297f58ea
I20250114 02:54:17.115394 1595 ts_tablet_manager.cc:1939] T 276942e5e54246c79f0d4f08297f58ea P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
I20250114 02:54:17.119707 1294 raft_consensus.cc:491] T 961b6a25a13a4fa2a732a6383dcd8c84 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:17.120137 1294 raft_consensus.cc:513] T 961b6a25a13a4fa2a732a6383dcd8c84 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:17.122257 1294 leader_election.cc:290] T 961b6a25a13a4fa2a732a6383dcd8c84 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:17.124454 1099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "961b6a25a13a4fa2a732a6383dcd8c84" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:17.125123 1099 raft_consensus.cc:2463] T 961b6a25a13a4fa2a732a6383dcd8c84 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 0.
I20250114 02:54:17.125952 1593 tablet_replica.cc:331] T ed2d137579b8467da5eed4ea2f619a9a P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
W20250114 02:54:17.125896 1040 leader_election.cc:343] T 00080bea6c8e46e584ccba6dcb0e221b P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:17.126787 1593 raft_consensus.cc:2238] T ed2d137579b8467da5eed4ea2f619a9a P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250114 02:54:17.126850 456 catalog_manager.cc:4839] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet b4740f6b3a6741368ba68907ad065279 because the tablet was not found. No further retry: Not found: Tablet not found: b4740f6b3a6741368ba68907ad065279
I20250114 02:54:17.127259 1593 raft_consensus.cc:2267] T ed2d137579b8467da5eed4ea2f619a9a P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250114 02:54:17.127450 456 catalog_manager.cc:4839] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet 83b5b36a26e74668998a19fa0ddf7188 because the tablet was not found. No further retry: Not found: Tablet not found: 83b5b36a26e74668998a19fa0ddf7188
I20250114 02:54:17.128710 464 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet 961b6a25a13a4fa2a732a6383dcd8c84
I20250114 02:54:17.129902 464 catalog_manager.cc:6596] ReplaceTablet: tablet 961b6a25a13a4fa2a732a6383dcd8c84 deleted and replaced by tablet 99b940d872ce4781907517c40fe08781
I20250114 02:54:17.130124 1593 ts_tablet_manager.cc:1905] T ed2d137579b8467da5eed4ea2f619a9a P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:17.130797 464 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_ca7bc63288b64f51806ca0f2a9129ff6" } new_table_name: "default.Table_0a0bda55a4a9476c98d961822b8a19bd" modify_external_catalogs: true
I20250114 02:54:17.131705 816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00080bea6c8e46e584ccba6dcb0e221b" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:17.132366 816 raft_consensus.cc:3054] T 00080bea6c8e46e584ccba6dcb0e221b P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:17.136963 469 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_5e1c3ceefda2466da137a659035f9ae4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:17.138346 469 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5e1c3ceefda2466da137a659035f9ae4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:17.140581 937 tablet_service.cc:1514] Processing DeleteTablet for tablet 961b6a25a13a4fa2a732a6383dcd8c84 with delete_type TABLET_DATA_DELETED (replaced by tablet 99b940d872ce4781907517c40fe08781) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:17.147785 798 tablet_service.cc:1514] Processing DeleteTablet for tablet 961b6a25a13a4fa2a732a6383dcd8c84 with delete_type TABLET_DATA_DELETED (replaced by tablet 99b940d872ce4781907517c40fe08781) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:17.148000 753 leader_election.cc:304] T 961b6a25a13a4fa2a732a6383dcd8c84 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:17.148825 1294 raft_consensus.cc:2798] T 961b6a25a13a4fa2a732a6383dcd8c84 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:17.149158 1294 raft_consensus.cc:491] T 961b6a25a13a4fa2a732a6383dcd8c84 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:17.149082 1101 raft_consensus.cc:2463] T de4ca83a69704a368a25954cdc1b9392 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:17.149477 1294 raft_consensus.cc:3054] T 961b6a25a13a4fa2a732a6383dcd8c84 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:17.151356 1084 tablet_service.cc:1514] Processing DeleteTablet for tablet 961b6a25a13a4fa2a732a6383dcd8c84 with delete_type TABLET_DATA_DELETED (replaced by tablet 99b940d872ce4781907517c40fe08781) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:17.157452 1207 tablet_bootstrap.cc:492] T 9812754c610548d1b43084bbc94fc3e4 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:17.158342 898 leader_election.cc:304] T de4ca83a69704a368a25954cdc1b9392 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters: 819f9e3412e54d44abb11d8b3ca877c2
I20250114 02:54:17.159585 1597 raft_consensus.cc:2798] T de4ca83a69704a368a25954cdc1b9392 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:17.160070 1597 raft_consensus.cc:695] T de4ca83a69704a368a25954cdc1b9392 P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Becoming Leader. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Running, Role: LEADER
I20250114 02:54:17.160856 1597 consensus_queue.cc:237] T de4ca83a69704a368a25954cdc1b9392 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:17.163877 816 raft_consensus.cc:2463] T 00080bea6c8e46e584ccba6dcb0e221b P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1.
I20250114 02:54:17.165105 1042 leader_election.cc:304] T 00080bea6c8e46e584ccba6dcb0e221b P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters: 525198593c8e47afaee54e74267b87e3
I20250114 02:54:17.166286 1508 raft_consensus.cc:2798] T 00080bea6c8e46e584ccba6dcb0e221b P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:17.166767 1508 raft_consensus.cc:695] T 00080bea6c8e46e584ccba6dcb0e221b P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Becoming Leader. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Running, Role: LEADER
I20250114 02:54:17.166559 465 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_44d578b4433242d5b7d8a61f6e4213e7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:17.167943 465 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_44d578b4433242d5b7d8a61f6e4213e7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:17.171177 1595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 113) took 1297 ms. Trace:
I20250114 02:54:17.172724 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "961b6a25a13a4fa2a732a6383dcd8c84" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
W20250114 02:54:17.174180 751 leader_election.cc:343] T 961b6a25a13a4fa2a732a6383dcd8c84 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:17.167582 1508 consensus_queue.cc:237] T 00080bea6c8e46e584ccba6dcb0e221b P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:17.177794 1099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15704785f56e43a3af69f181c75877ce" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:17.178398 1099 raft_consensus.cc:3054] T 15704785f56e43a3af69f181c75877ce P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:17.171656 1595 rpcz_store.cc:276] 0114 02:54:15.873974 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:15.874142 (+ 168us) service_pool.cc:225] Handling call
0114 02:54:16.671925 (+797783us) ts_tablet_manager.cc:1145] Deleting tablet 276942e5e54246c79f0d4f08297f58ea
0114 02:54:16.671972 (+ 47us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:16.969766 (+297794us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:17.114039 (+144273us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:17.171103 (+ 57064us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":797100}
I20250114 02:54:17.191618 1595 tablet_replica.cc:331] T 268a12df225e4ba59d4a84c2869d4661 P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:17.192595 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet 276942e5e54246c79f0d4f08297f58ea (table default.Table_b6330aba5846413c915c50dc4c6e5f11 [id=c3d72f049d9849ab86e252c47e5d24e2]) successfully deleted
I20250114 02:54:17.192739 1593 ts_tablet_manager.cc:1918] T ed2d137579b8467da5eed4ea2f619a9a P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:17.193109 1593 log.cc:1198] T ed2d137579b8467da5eed4ea2f619a9a P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/ed2d137579b8467da5eed4ea2f619a9a
I20250114 02:54:17.193775 1593 ts_tablet_manager.cc:1939] T ed2d137579b8467da5eed4ea2f619a9a P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:17.195399 1595 raft_consensus.cc:2238] T 268a12df225e4ba59d4a84c2869d4661 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:17.195853 1595 raft_consensus.cc:2267] T 268a12df225e4ba59d4a84c2869d4661 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:17.196051 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet ed2d137579b8467da5eed4ea2f619a9a (table default.Table_4c3843233dda407395ab9652d5870bc6 [id=0d173fdb502f482fb86b6d3e5fdc66f1]) successfully deleted
I20250114 02:54:17.197477 1593 tablet_replica.cc:331] T c20aca761313498193e1d79c10705f8f P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:17.198196 1593 raft_consensus.cc:2238] T c20aca761313498193e1d79c10705f8f P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:17.199374 471 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 4bb9b429af92446bbe13894d5dfe4c47 (table default.Table_acb5bf9354be440bbf6034cf1c535dab [id=22b7edcf4b7e4c31b3c23dc44ccf028e]). Expected version 1 got 0
I20250114 02:54:17.200337 1593 raft_consensus.cc:2267] T c20aca761313498193e1d79c10705f8f P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:17.200490 471 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 179248cb5acf4740b268e0ec8b2852f9 (table default.Table_bd405598fe054b718a8d6fb5dd875f0e [id=5745501ecd7b4cecaca3d35502a6cc40]). Expected version 1 got 0
I20250114 02:54:17.201189 471 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 8b4bc535c42e43889401f14b6d59659e (table default.Table_d90b2995b13b4a8fb7e50cc4b79cbf84 [id=f8e40e0aa44b4157ac3438d11685b2e7]). Expected version 1 got 0
I20250114 02:54:17.202499 1593 ts_tablet_manager.cc:1905] T c20aca761313498193e1d79c10705f8f P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:17.212198 468 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 179248cb5acf4740b268e0ec8b2852f9 (table default.Table_bd405598fe054b718a8d6fb5dd875f0e [id=5745501ecd7b4cecaca3d35502a6cc40]). Expected version 1 got 0
I20250114 02:54:17.214653 468 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 4bb9b429af92446bbe13894d5dfe4c47 (table default.Table_acb5bf9354be440bbf6034cf1c535dab [id=22b7edcf4b7e4c31b3c23dc44ccf028e]). Expected version 1 got 0
I20250114 02:54:17.215543 468 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet a3fdab330d574c0283ac58bc9e450b87 (table default.Table_ec0eefb64458440ebd57a05a7ad0bb9a [id=b6874c22fe0d4151a7c8ea061f685938]). Expected version 1 got 0
I20250114 02:54:17.223956 1099 raft_consensus.cc:2463] T 15704785f56e43a3af69f181c75877ce P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:17.215692 795 tablet_service.cc:1514] Processing DeleteTablet for tablet 99b8acb1fe594bf3825c7b9121018db6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:16 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:17.226576 1594 raft_consensus.cc:2267] T 9589afd0094545c1801b7a0b5eb8698f P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:17.228286 898 leader_election.cc:304] T 15704785f56e43a3af69f181c75877ce P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters: 819f9e3412e54d44abb11d8b3ca877c2
I20250114 02:54:17.229048 1601 raft_consensus.cc:2798] T 15704785f56e43a3af69f181c75877ce P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:17.229537 1207 tablet_bootstrap.cc:654] T 9812754c610548d1b43084bbc94fc3e4 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:17.229466 1601 raft_consensus.cc:695] T 15704785f56e43a3af69f181c75877ce P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Becoming Leader. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Running, Role: LEADER
I20250114 02:54:17.230139 1601 consensus_queue.cc:237] T 15704785f56e43a3af69f181c75877ce P 525198593c8e47afaee54e74267b87e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:17.242805 1593 ts_tablet_manager.cc:1918] T c20aca761313498193e1d79c10705f8f P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:17.243186 1593 log.cc:1198] T c20aca761313498193e1d79c10705f8f P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/c20aca761313498193e1d79c10705f8f
I20250114 02:54:17.218839 1294 raft_consensus.cc:513] T 961b6a25a13a4fa2a732a6383dcd8c84 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:17.245956 1597 consensus_queue.cc:1035] T a3fdab330d574c0283ac58bc9e450b87 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:17.197710 1595 ts_tablet_manager.cc:1905] T 268a12df225e4ba59d4a84c2869d4661 P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:17.247570 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "961b6a25a13a4fa2a732a6383dcd8c84" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
I20250114 02:54:17.249384 470 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:33540 (request call id 162) took 1357 ms. Trace:
W20250114 02:54:17.250053 751 leader_election.cc:343] T 961b6a25a13a4fa2a732a6383dcd8c84 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:17.253300 1593 ts_tablet_manager.cc:1939] T c20aca761313498193e1d79c10705f8f P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:17.249666 470 rpcz_store.cc:276] 0114 02:54:15.892473 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:15.925880 (+ 33407us) service_pool.cc:225] Handling call
0114 02:54:16.012677 (+ 86797us) spinlock_profiling.cc:243] Waited 84.8 ms on lock 0x7b54000316d4. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616cad50 00007f646174df2a 00007f64633f2dec 00007f64634ec986 00007f6461259644 00007f64612595d2 00007f6461259501 00007f646125947c 00007f6461258732 00007f645f512dfc 00007f645f512236
0114 02:54:16.024404 (+ 11727us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0114 02:54:16.025027 (+ 623us) catalog_manager.cc:3627] Apply alter schema
0114 02:54:16.025559 (+ 532us) catalog_manager.cc:3659] Acquired catalog manager lock
0114 02:54:16.025758 (+ 199us) catalog_manager.cc:3817] Updating metadata on disk
0114 02:54:16.672335 (+646577us) catalog_manager.cc:3848] Committing alterations to in-memory state
0114 02:54:17.249330 (+576995us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:16.223186 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:16.223495 (+ 309us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:16.223517 (+ 22us) write_op.cc:435] Acquired schema lock
0114 02:54:16.223529 (+ 12us) tablet.cc:659] Decoding operations
0114 02:54:16.223733 (+ 204us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:16.223763 (+ 30us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:16.223786 (+ 23us) tablet.cc:682] Acquiring locks for 1 operations
0114 02:54:16.223968 (+ 182us) tablet.cc:698] Row locks acquired
0114 02:54:16.223987 (+ 19us) write_op.cc:260] PREPARE: finished
0114 02:54:16.520671 (+296684us) write_op.cc:270] Start()
0114 02:54:16.520778 (+ 107us) write_op.cc:276] Timestamp: P: 1736823256223877 usec, L: 0
0114 02:54:16.520804 (+ 26us) op_driver.cc:348] REPLICATION: starting
0114 02:54:16.521483 (+ 679us) log.cc:844] Serialized 405 byte log entry
0114 02:54:16.669319 (+147836us) op_driver.cc:464] REPLICATION: finished
0114 02:54:16.670107 (+ 788us) write_op.cc:301] APPLY: starting
0114 02:54:16.670181 (+ 74us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:16.670292 (+ 111us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:16.670398 (+ 106us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:16.670614 (+ 216us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:16.670745 (+ 131us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:16.670782 (+ 37us) write_op.cc:312] APPLY: finished
0114 02:54:16.670948 (+ 166us) log.cc:844] Serialized 23 byte log entry
0114 02:54:16.671090 (+ 142us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:16.671162 (+ 72us) write_op.cc:454] Released schema lock
0114 02:54:16.671189 (+ 27us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:16.671383 (+ 194us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:16.671407 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":194976128,"child_traces":[["op",{"apply.queue_time_us":623,"apply.run_cpu_time_us":1256,"apply.run_wall_time_us":1256,"num_ops":1,"prepare.queue_time_us":196334,"prepare.run_cpu_time_us":1947,"prepare.run_wall_time_us":299628,"replication_time_us":148425}]]}
I20250114 02:54:17.255714 1593 tablet_replica.cc:331] T b63dd90f60a04465b948171e74888e79 P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:17.256423 1593 raft_consensus.cc:2238] T b63dd90f60a04465b948171e74888e79 P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Raft consensus shutting down.
I20250114 02:54:17.256449 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet c20aca761313498193e1d79c10705f8f (table default.Table_6a4fe39298514187b65def5841d3a992 [id=5c5a7974ad2f4e58ba5c47a6eeaaac62]) successfully deleted
I20250114 02:54:17.257172 1593 raft_consensus.cc:2267] T b63dd90f60a04465b948171e74888e79 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:17.257755 466 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet 66bfa92db9874750b3c340e31a06c412
I20250114 02:54:17.258977 466 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:33540 (request call id 165) took 1210 ms. Trace:
I20250114 02:54:17.259250 466 rpcz_store.cc:276] 0114 02:54:16.048739 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:16.230371 (+181632us) service_pool.cc:225] Handling call
0114 02:54:16.284191 (+ 53820us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0114 02:54:16.287841 (+ 3650us) catalog_manager.cc:2753] Modifying in-memory table state
0114 02:54:16.287986 (+ 145us) catalog_manager.cc:2762] Locking tablets
0114 02:54:16.752875 (+464889us) catalog_manager.cc:2776] Removing table and tablets from system table
0114 02:54:17.016905 (+264030us) catalog_manager.cc:2794] Removing table from by-name map
0114 02:54:17.017107 (+ 202us) catalog_manager.cc:2810] Committing in-memory state
0114 02:54:17.257675 (+240568us) catalog_manager.cc:2814] Aborting table tasks
0114 02:54:17.258925 (+ 1250us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:16.924754 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:16.925102 (+ 348us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:16.925124 (+ 22us) write_op.cc:435] Acquired schema lock
0114 02:54:16.925136 (+ 12us) tablet.cc:659] Decoding operations
0114 02:54:16.925435 (+ 299us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:16.925469 (+ 34us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:16.925490 (+ 21us) tablet.cc:682] Acquiring locks for 2 operations
0114 02:54:16.925713 (+ 223us) tablet.cc:698] Row locks acquired
0114 02:54:16.925725 (+ 12us) write_op.cc:260] PREPARE: finished
0114 02:54:16.925816 (+ 91us) write_op.cc:270] Start()
0114 02:54:16.925889 (+ 73us) write_op.cc:276] Timestamp: P: 1736823256925748 usec, L: 0
0114 02:54:16.925903 (+ 14us) op_driver.cc:348] REPLICATION: starting
0114 02:54:17.005611 (+ 79708us) log.cc:844] Serialized 728 byte log entry
0114 02:54:17.006425 (+ 814us) spinlock_profiling.cc:243] Waited 79.1 ms on lock 0x7b54000316d4. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616ca6d1 00007f6461741198 00007f6461bad3b1 00007f6461bacb88 00007f6461bb1821 00007f6461bb17d9 00007f6461bb1769 00007f6461bb1731 00007f6461bb095d 000000000059ea04 000000000059e839
0114 02:54:17.014910 (+ 8485us) op_driver.cc:464] REPLICATION: finished
0114 02:54:17.015361 (+ 451us) write_op.cc:301] APPLY: starting
0114 02:54:17.015417 (+ 56us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:17.015514 (+ 97us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:17.015534 (+ 20us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:17.015764 (+ 230us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:17.015898 (+ 134us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0114 02:54:17.015936 (+ 38us) write_op.cc:312] APPLY: finished
0114 02:54:17.016126 (+ 190us) log.cc:844] Serialized 29 byte log entry
0114 02:54:17.016275 (+ 149us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:17.016343 (+ 68us) write_op.cc:454] Released schema lock
0114 02:54:17.016373 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:17.016593 (+ 220us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:17.016617 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":317,"apply.run_cpu_time_us":1195,"apply.run_wall_time_us":1194,"num_ops":2,"prepare.queue_time_us":170306,"prepare.run_cpu_time_us":2961,"prepare.run_wall_time_us":81938,"raft.queue_time_us":605,"raft.run_cpu_time_us":569,"raft.run_wall_time_us":570,"replication_time_us":88907,"spinlock_wait_cycles":181833472}]]}
I20250114 02:54:17.264855 1593 ts_tablet_manager.cc:1905] T b63dd90f60a04465b948171e74888e79 P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:17.265786 798 tablet_service.cc:1514] Processing DeleteTablet for tablet 66bfa92db9874750b3c340e31a06c412 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:16 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:17.267061 1080 tablet_service.cc:1514] Processing DeleteTablet for tablet 66bfa92db9874750b3c340e31a06c412 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:16 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:17.269490 1205 raft_consensus.cc:357] T 00080bea6c8e46e584ccba6dcb0e221b P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:17.270522 1205 raft_consensus.cc:383] T 00080bea6c8e46e584ccba6dcb0e221b P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:17.270776 1205 raft_consensus.cc:738] T 00080bea6c8e46e584ccba6dcb0e221b P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:17.271319 1205 consensus_queue.cc:260] T 00080bea6c8e46e584ccba6dcb0e221b P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:17.278439 1594 ts_tablet_manager.cc:1905] T 9589afd0094545c1801b7a0b5eb8698f P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:17.290846 795 tablet_service.cc:1514] Processing DeleteTablet for tablet b4740f6b3a6741368ba68907ad065279 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:14 UTC) from {username='slave'} at 127.0.0.1:53078
W20250114 02:54:17.295531 1040 consensus_peers.cc:487] T ed2d137579b8467da5eed4ea2f619a9a P 90d6c2c58bb54e0eafb365625ac33871 -> Peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Couldn't send request to peer 525198593c8e47afaee54e74267b87e3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ed2d137579b8467da5eed4ea2f619a9a. This is attempt 1: this message will repeat every 5th retry.
I20250114 02:54:17.300431 1654 tablet.cc:1721] T 179248cb5acf4740b268e0ec8b2852f9 P 819f9e3412e54d44abb11d8b3ca877c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250114 02:54:17.296248 1593 ts_tablet_manager.cc:1918] T b63dd90f60a04465b948171e74888e79 P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250114 02:54:17.304159 1086 tablet_service.cc:1514] Processing DeleteTablet for tablet b4740f6b3a6741368ba68907ad065279 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:14 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:17.304610 1593 log.cc:1198] T b63dd90f60a04465b948171e74888e79 P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/b63dd90f60a04465b948171e74888e79
I20250114 02:54:17.305379 1593 ts_tablet_manager.cc:1939] T b63dd90f60a04465b948171e74888e79 P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:17.305409 1086 tablet_service.cc:1514] Processing DeleteTablet for tablet 290ad07ab8be4791a39194909e000843 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:13 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:17.308421 1080 tablet_service.cc:1514] Processing DeleteTablet for tablet 276942e5e54246c79f0d4f08297f58ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:14 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:17.316804 938 tablet_service.cc:1514] Processing DeleteTablet for tablet 66bfa92db9874750b3c340e31a06c412 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:16 UTC) from {username='slave'} at 127.0.0.1:55068
W20250114 02:54:17.321205 467 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.67:34825 (request call id 38) took 1771 ms (client timeout 1000 ms). Trace:
I20250114 02:54:17.321455 1655 tablet.cc:1721] T 179248cb5acf4740b268e0ec8b2852f9 P 525198593c8e47afaee54e74267b87e3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250114 02:54:17.321578 467 rpcz_store.cc:269] 0114 02:54:15.550129 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:15.679912 (+129783us) service_pool.cc:225] Handling call
0114 02:54:15.731168 (+ 51256us) spinlock_profiling.cc:243] Waited 20.8 ms on lock 0x7b54000316d4. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616cad50 00007f646174df2a 00007f64633f2dec 00007f64634eadee 00007f6461248fc4 00007f6461248f52 00007f6461248e81 00007f6461248dfc 00007f64612480b2 00007f645f512dfc 00007f645f512236
0114 02:54:17.321140 (+1589972us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22675,"reported_tablets":18,"spinlock_wait_cycles":47922560}
I20250114 02:54:17.346143 1206 tablet_bootstrap.cc:492] T db24bf3db61941ba9b5e0662db23e748 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:17.346513 1206 ts_tablet_manager.cc:1397] T db24bf3db61941ba9b5e0662db23e748 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.269s user 0.010s sys 0.004s
I20250114 02:54:17.348482 1206 raft_consensus.cc:357] T db24bf3db61941ba9b5e0662db23e748 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:17.349138 1206 raft_consensus.cc:383] T db24bf3db61941ba9b5e0662db23e748 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:17.349366 1206 raft_consensus.cc:738] T db24bf3db61941ba9b5e0662db23e748 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:17.349866 1206 consensus_queue.cc:260] T db24bf3db61941ba9b5e0662db23e748 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:17.355082 1549 raft_consensus.cc:491] T 7ecf782bb3d54528aa996abca1ea3a15 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:17.355486 1549 raft_consensus.cc:513] T 7ecf782bb3d54528aa996abca1ea3a15 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:17.360970 961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ecf782bb3d54528aa996abca1ea3a15" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:17.361838 961 raft_consensus.cc:2388] T 7ecf782bb3d54528aa996abca1ea3a15 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 819f9e3412e54d44abb11d8b3ca877c2 in current term 1: Already voted for candidate 90d6c2c58bb54e0eafb365625ac33871 in this term.
I20250114 02:54:17.364315 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet b63dd90f60a04465b948171e74888e79 (table default.Table_1097f09fb10e441fb639f8d7aa12d487 [id=d8c9691818d042d7b49475fa009c4ca1]) successfully deleted
W20250114 02:54:17.365343 456 catalog_manager.cc:4852] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet 99b8acb1fe594bf3825c7b9121018db6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 99b8acb1fe594bf3825c7b9121018db6 already in progress: creating tablet
W20250114 02:54:17.365950 456 catalog_manager.cc:4852] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet 961b6a25a13a4fa2a732a6383dcd8c84 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 961b6a25a13a4fa2a732a6383dcd8c84 already in progress: creating tablet
I20250114 02:54:17.366055 1507 raft_consensus.cc:491] T db24bf3db61941ba9b5e0662db23e748 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 02:54:17.366515 456 catalog_manager.cc:4852] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet 66bfa92db9874750b3c340e31a06c412 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 66bfa92db9874750b3c340e31a06c412 already in progress: creating tablet
I20250114 02:54:17.366463 1507 raft_consensus.cc:513] T db24bf3db61941ba9b5e0662db23e748 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:17.368314 1507 leader_election.cc:290] T db24bf3db61941ba9b5e0662db23e748 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:17.369602 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db24bf3db61941ba9b5e0662db23e748" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:17.370137 818 raft_consensus.cc:2463] T db24bf3db61941ba9b5e0662db23e748 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 0.
I20250114 02:54:17.373039 1042 leader_election.cc:304] T db24bf3db61941ba9b5e0662db23e748 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:17.373857 1507 raft_consensus.cc:2798] T db24bf3db61941ba9b5e0662db23e748 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:17.374203 1507 raft_consensus.cc:491] T db24bf3db61941ba9b5e0662db23e748 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:17.374704 1507 raft_consensus.cc:3054] T db24bf3db61941ba9b5e0662db23e748 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:17.380038 961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db24bf3db61941ba9b5e0662db23e748" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
W20250114 02:54:17.382565 1040 leader_election.cc:343] T db24bf3db61941ba9b5e0662db23e748 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:17.384443 1507 raft_consensus.cc:513] T db24bf3db61941ba9b5e0662db23e748 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:17.387130 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db24bf3db61941ba9b5e0662db23e748" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
W20250114 02:54:17.388217 1040 leader_election.cc:343] T db24bf3db61941ba9b5e0662db23e748 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:17.393656 1206 ts_tablet_manager.cc:1428] T db24bf3db61941ba9b5e0662db23e748 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.047s user 0.006s sys 0.001s
I20250114 02:54:17.394132 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db24bf3db61941ba9b5e0662db23e748" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:17.394482 1206 tablet_bootstrap.cc:492] T de4ca83a69704a368a25954cdc1b9392 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:17.394631 818 raft_consensus.cc:3054] T db24bf3db61941ba9b5e0662db23e748 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:17.398296 1101 raft_consensus.cc:3054] T 290ad07ab8be4791a39194909e000843 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:17.404156 818 raft_consensus.cc:2463] T db24bf3db61941ba9b5e0662db23e748 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1.
I20250114 02:54:17.405450 1042 leader_election.cc:304] T db24bf3db61941ba9b5e0662db23e748 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters: 525198593c8e47afaee54e74267b87e3
I20250114 02:54:17.406168 1506 raft_consensus.cc:2798] T db24bf3db61941ba9b5e0662db23e748 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:17.406695 1506 raft_consensus.cc:695] T db24bf3db61941ba9b5e0662db23e748 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Becoming Leader. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Running, Role: LEADER
I20250114 02:54:17.407495 1507 leader_election.cc:290] T db24bf3db61941ba9b5e0662db23e748 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Requested vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:17.408874 466 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_eb7878bdce82445c97f6ce665eb52d8e" } modify_external_catalogs: true
I20250114 02:54:17.407621 1506 consensus_queue.cc:237] T db24bf3db61941ba9b5e0662db23e748 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:17.410017 470 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_395ca40035c8410ca1ea6b0e1d54bf37"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:17.413419 470 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_395ca40035c8410ca1ea6b0e1d54bf37 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:17.411319 467 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_53a42668384c4379998f7d6e5c8b36e7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:17.414790 467 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_53a42668384c4379998f7d6e5c8b36e7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:17.415138 1206 tablet_bootstrap.cc:654] T de4ca83a69704a368a25954cdc1b9392 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:17.415684 466 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_eb7878bdce82445c97f6ce665eb52d8e" } modify_external_catalogs: true
I20250114 02:54:17.424484 795 tablet_service.cc:1514] Processing DeleteTablet for tablet 83b5b36a26e74668998a19fa0ddf7188 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:12 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:17.426651 1595 ts_tablet_manager.cc:1918] T 268a12df225e4ba59d4a84c2869d4661 P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:17.427084 1595 log.cc:1198] T 268a12df225e4ba59d4a84c2869d4661 P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/268a12df225e4ba59d4a84c2869d4661
I20250114 02:54:17.427955 1595 ts_tablet_manager.cc:1939] T 268a12df225e4ba59d4a84c2869d4661 P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
I20250114 02:54:17.434211 1595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 114) took 1555 ms. Trace:
I20250114 02:54:17.434545 1595 rpcz_store.cc:276] 0114 02:54:15.878502 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:15.878618 (+ 116us) service_pool.cc:225] Handling call
0114 02:54:17.191462 (+1312844us) ts_tablet_manager.cc:1145] Deleting tablet 268a12df225e4ba59d4a84c2869d4661
0114 02:54:17.191505 (+ 43us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:17.359031 (+167526us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:17.426398 (+ 67367us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:17.434145 (+ 7747us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1312322}
I20250114 02:54:17.435771 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet 268a12df225e4ba59d4a84c2869d4661 (table default.Table_a1731c2cc5964c3aaddb43ec22096aad [id=67870401767c48be838c7163a9ba8beb]) successfully deleted
I20250114 02:54:17.448565 469 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_7176743e4429467481ed67254643b9f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:17.449982 469 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7176743e4429467481ed67254643b9f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250114 02:54:17.452859 464 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet edc69b66bea54625a4b15867b75986b7 has no consensus state
I20250114 02:54:17.465312 1206 tablet_bootstrap.cc:492] T de4ca83a69704a368a25954cdc1b9392 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:17.468248 1206 ts_tablet_manager.cc:1397] T de4ca83a69704a368a25954cdc1b9392 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.074s user 0.011s sys 0.003s
I20250114 02:54:17.469345 1205 ts_tablet_manager.cc:1428] T 00080bea6c8e46e584ccba6dcb0e221b P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.375s user 0.006s sys 0.000s
I20250114 02:54:17.470217 1205 tablet_bootstrap.cc:492] T db24bf3db61941ba9b5e0662db23e748 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:17.480950 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ecf782bb3d54528aa996abca1ea3a15" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
W20250114 02:54:17.486464 1040 consensus_peers.cc:487] T c20aca761313498193e1d79c10705f8f P 90d6c2c58bb54e0eafb365625ac33871 -> Peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Couldn't send request to peer 525198593c8e47afaee54e74267b87e3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c20aca761313498193e1d79c10705f8f. This is attempt 1: this message will repeat every 5th retry.
I20250114 02:54:17.475724 795 tablet_service.cc:1514] Processing DeleteTablet for tablet ed2d137579b8467da5eed4ea2f619a9a with delete_type TABLET_DATA_DELETED (replaced by tablet 0304df55431e42e1a999ed17dc35e6c0) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:17.488482 1601 consensus_queue.cc:1035] T 290ad07ab8be4791a39194909e000843 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:17.493479 1595 tablet_replica.cc:331] T ed2d137579b8467da5eed4ea2f619a9a P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:17.495173 1595 raft_consensus.cc:2238] T ed2d137579b8467da5eed4ea2f619a9a P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Raft consensus shutting down.
I20250114 02:54:17.496104 1595 raft_consensus.cc:2267] T ed2d137579b8467da5eed4ea2f619a9a P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:17.498149 1595 ts_tablet_manager.cc:1905] T ed2d137579b8467da5eed4ea2f619a9a P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:17.498586 1101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "961b6a25a13a4fa2a732a6383dcd8c84" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:17.499107 1101 raft_consensus.cc:3054] T 961b6a25a13a4fa2a732a6383dcd8c84 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:17.505290 1206 raft_consensus.cc:357] T de4ca83a69704a368a25954cdc1b9392 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:17.483650 753 leader_election.cc:304] T 7ecf782bb3d54528aa996abca1ea3a15 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:17.476666 1294 leader_election.cc:290] T 961b6a25a13a4fa2a732a6383dcd8c84 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Requested vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:17.513309 1294 raft_consensus.cc:3054] T 7ecf782bb3d54528aa996abca1ea3a15 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:17.513779 1206 raft_consensus.cc:383] T de4ca83a69704a368a25954cdc1b9392 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:17.514629 1206 raft_consensus.cc:738] T de4ca83a69704a368a25954cdc1b9392 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:17.516706 1206 consensus_queue.cc:260] T de4ca83a69704a368a25954cdc1b9392 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:17.523437 1206 ts_tablet_manager.cc:1428] T de4ca83a69704a368a25954cdc1b9392 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.036s user 0.006s sys 0.000s
I20250114 02:54:17.525821 1206 tablet_bootstrap.cc:492] T 15704785f56e43a3af69f181c75877ce P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:17.512634 1549 leader_election.cc:290] T 7ecf782bb3d54528aa996abca1ea3a15 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:17.512086 795 tablet_service.cc:1514] Processing DeleteTablet for tablet 268a12df225e4ba59d4a84c2869d4661 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:10 UTC) from {username='slave'} at 127.0.0.1:53078
W20250114 02:54:17.553357 898 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.65:39397, user_credentials={real_user=slave}} blocked reactor thread for 63196.6us
I20250114 02:54:17.560614 1206 tablet_bootstrap.cc:654] T 15704785f56e43a3af69f181c75877ce P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:17.562485 464 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_569df7e8f022465c96cd2a326e905c72" } new_table_name: "default.Table_b624816662024c55a5f5ef5830b20b7c" modify_external_catalogs: true
I20250114 02:54:17.565430 465 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_5d3b333f1d99463b8ad3b4318c8f29df" } modify_external_catalogs: true
I20250114 02:54:17.561965 1654 tablet.cc:1721] T 4bb9b429af92446bbe13894d5dfe4c47 P 819f9e3412e54d44abb11d8b3ca877c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250114 02:54:17.574819 1207 tablet_bootstrap.cc:492] T 9812754c610548d1b43084bbc94fc3e4 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:17.575197 1207 ts_tablet_manager.cc:1397] T 9812754c610548d1b43084bbc94fc3e4 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.418s user 0.006s sys 0.003s
I20250114 02:54:17.578286 465 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_5d3b333f1d99463b8ad3b4318c8f29df" } modify_external_catalogs: true
W20250114 02:54:17.580613 1656 log.cc:926] Time spent T 4bb9b429af92446bbe13894d5dfe4c47 P 819f9e3412e54d44abb11d8b3ca877c2: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
W20250114 02:54:17.586015 471 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.65:49805 (request call id 41) took 2055 ms (client timeout 1000 ms). Trace:
W20250114 02:54:17.586357 471 rpcz_store.cc:269] 0114 02:54:15.530819 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:15.675316 (+144497us) service_pool.cc:225] Handling call
0114 02:54:15.724813 (+ 49497us) spinlock_profiling.cc:243] Waited 37.2 ms on lock 0x7b54000316d4. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616cad50 00007f646174df2a 00007f64633f2dec 00007f64634eadee 00007f6461248fc4 00007f6461248f52 00007f6461248e81 00007f6461248dfc 00007f64612480b2 00007f645f512dfc 00007f645f512236
0114 02:54:17.585954 (+1861141us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":12,"spinlock_wait_cycles":85593984}
I20250114 02:54:17.595865 795 tablet_service.cc:1514] Processing DeleteTablet for tablet c20aca761313498193e1d79c10705f8f with delete_type TABLET_DATA_DELETED (replaced by tablet bc46815b4803418c95ea9c039a4fae88) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:17.600827 1663 tablet.cc:1721] T 179248cb5acf4740b268e0ec8b2852f9 P 90d6c2c58bb54e0eafb365625ac33871: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250114 02:54:17.626214 1663 tablet.cc:1721] T 4bb9b429af92446bbe13894d5dfe4c47 P 90d6c2c58bb54e0eafb365625ac33871: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250114 02:54:17.654759 456 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet edc69b66bea54625a4b15867b75986b7 (table default.Table_0a0bda55a4a9476c98d961822b8a19bd [id=fb2be5c1ca9444a197f3315ea9c59b09], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet edc69b66bea54625a4b15867b75986b7 has no consensus state
I20250114 02:54:17.658360 1206 tablet_bootstrap.cc:492] T 15704785f56e43a3af69f181c75877ce P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:17.658756 1206 ts_tablet_manager.cc:1397] T 15704785f56e43a3af69f181c75877ce P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.133s user 0.014s sys 0.006s
I20250114 02:54:17.669044 1206 raft_consensus.cc:357] T 15704785f56e43a3af69f181c75877ce P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:17.669618 1206 raft_consensus.cc:383] T 15704785f56e43a3af69f181c75877ce P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:17.669802 1206 raft_consensus.cc:738] T 15704785f56e43a3af69f181c75877ce P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:17.673825 1206 consensus_queue.cc:260] T 15704785f56e43a3af69f181c75877ce P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:17.674252 1618 consensus_queue.cc:1035] T a3fdab330d574c0283ac58bc9e450b87 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:17.680959 1207 raft_consensus.cc:357] T 9812754c610548d1b43084bbc94fc3e4 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:17.699630 1206 ts_tablet_manager.cc:1428] T 15704785f56e43a3af69f181c75877ce P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.041s user 0.002s sys 0.003s
I20250114 02:54:17.713572 1206 tablet_bootstrap.cc:492] T fcb58a05f980460884ed81a30a5d59c2 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:17.719933 1206 tablet_bootstrap.cc:654] T fcb58a05f980460884ed81a30a5d59c2 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
W20250114 02:54:17.747675 898 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.67:45749, user_credentials={real_user=slave}} blocked reactor thread for 71880.2us
I20250114 02:54:17.748637 1601 consensus_queue.cc:1035] T 9bfd8fd6d2104fb4a6484cb8c98981bd P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:17.744432 1505 consensus_queue.cc:1035] T 7ecf782bb3d54528aa996abca1ea3a15 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:17.756076 1206 tablet_bootstrap.cc:492] T fcb58a05f980460884ed81a30a5d59c2 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:17.756512 1206 ts_tablet_manager.cc:1397] T fcb58a05f980460884ed81a30a5d59c2 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.045s user 0.006s sys 0.003s
I20250114 02:54:17.760331 1207 raft_consensus.cc:383] T 9812754c610548d1b43084bbc94fc3e4 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:17.764505 1205 tablet_bootstrap.cc:654] T db24bf3db61941ba9b5e0662db23e748 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:17.768536 1207 raft_consensus.cc:738] T 9812754c610548d1b43084bbc94fc3e4 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:17.769179 1206 raft_consensus.cc:357] T fcb58a05f980460884ed81a30a5d59c2 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:17.769742 1206 raft_consensus.cc:383] T fcb58a05f980460884ed81a30a5d59c2 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:17.769953 1206 raft_consensus.cc:738] T fcb58a05f980460884ed81a30a5d59c2 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:17.770432 1206 consensus_queue.cc:260] T fcb58a05f980460884ed81a30a5d59c2 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
W20250114 02:54:17.776386 455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.107.124:39105: connect: Connection refused (error 111) [suppressed 6 similar messages]
I20250114 02:54:17.780714 1206 ts_tablet_manager.cc:1428] T fcb58a05f980460884ed81a30a5d59c2 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.024s user 0.005s sys 0.001s
I20250114 02:54:17.792486 1595 ts_tablet_manager.cc:1918] T ed2d137579b8467da5eed4ea2f619a9a P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:17.821177 1601 consensus_queue.cc:1035] T 290ad07ab8be4791a39194909e000843 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:17.831710 1667 tablet.cc:1721] T 4bb9b429af92446bbe13894d5dfe4c47 P 525198593c8e47afaee54e74267b87e3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250114 02:54:17.779803 1294 raft_consensus.cc:2743] T 7ecf782bb3d54528aa996abca1ea3a15 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:17.874495 1206 tablet_bootstrap.cc:492] T 99b8acb1fe594bf3825c7b9121018db6 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:17.882699 1206 tablet_bootstrap.cc:654] T 99b8acb1fe594bf3825c7b9121018db6 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:17.886159 1294 raft_consensus.cc:491] T 9812754c610548d1b43084bbc94fc3e4 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:17.886663 1294 raft_consensus.cc:513] T 9812754c610548d1b43084bbc94fc3e4 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:17.890118 1294 leader_election.cc:290] T 9812754c610548d1b43084bbc94fc3e4 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:17.900621 1618 consensus_queue.cc:1035] T 9bfd8fd6d2104fb4a6484cb8c98981bd P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:17.904891 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9812754c610548d1b43084bbc94fc3e4" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:17.910710 1595 log.cc:1198] T ed2d137579b8467da5eed4ea2f619a9a P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/ed2d137579b8467da5eed4ea2f619a9a
W20250114 02:54:17.914335 751 leader_election.cc:343] T 9812754c610548d1b43084bbc94fc3e4 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:17.915573 1549 raft_consensus.cc:491] T 15704785f56e43a3af69f181c75877ce P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:17.915794 1595 ts_tablet_manager.cc:1939] T ed2d137579b8467da5eed4ea2f619a9a P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
I20250114 02:54:17.916075 1549 raft_consensus.cc:513] T 15704785f56e43a3af69f181c75877ce P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:17.845494 1101 raft_consensus.cc:2463] T 961b6a25a13a4fa2a732a6383dcd8c84 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 1.
I20250114 02:54:17.917493 753 leader_election.cc:304] T 961b6a25a13a4fa2a732a6383dcd8c84 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters: 525198593c8e47afaee54e74267b87e3
I20250114 02:54:17.917898 1549 leader_election.cc:290] T 15704785f56e43a3af69f181c75877ce P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:17.918522 1549 raft_consensus.cc:2798] T 961b6a25a13a4fa2a732a6383dcd8c84 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:17.863448 1505 consensus_queue.cc:1035] T 7ecf782bb3d54528aa996abca1ea3a15 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:17.889698 1102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9812754c610548d1b43084bbc94fc3e4" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:17.919330 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15704785f56e43a3af69f181c75877ce" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:17.920002 1106 raft_consensus.cc:2388] T 15704785f56e43a3af69f181c75877ce P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 819f9e3412e54d44abb11d8b3ca877c2 in current term 1: Already voted for candidate 525198593c8e47afaee54e74267b87e3 in this term.
I20250114 02:54:17.920192 1549 raft_consensus.cc:695] T 961b6a25a13a4fa2a732a6383dcd8c84 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 LEADER]: Becoming Leader. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Running, Role: LEADER
I20250114 02:54:17.921066 1549 consensus_queue.cc:237] T 961b6a25a13a4fa2a732a6383dcd8c84 P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:17.931921 961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15704785f56e43a3af69f181c75877ce" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:17.772095 1207 consensus_queue.cc:260] T 9812754c610548d1b43084bbc94fc3e4 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:17.947780 1102 raft_consensus.cc:2463] T 9812754c610548d1b43084bbc94fc3e4 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 0.
I20250114 02:54:17.950345 751 leader_election.cc:304] T 15704785f56e43a3af69f181c75877ce P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:17.951243 753 leader_election.cc:304] T 9812754c610548d1b43084bbc94fc3e4 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters: 525198593c8e47afaee54e74267b87e3
I20250114 02:54:17.952049 1652 raft_consensus.cc:2798] T 9812754c610548d1b43084bbc94fc3e4 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:17.952374 1652 raft_consensus.cc:491] T 9812754c610548d1b43084bbc94fc3e4 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:17.952642 1652 raft_consensus.cc:3054] T 9812754c610548d1b43084bbc94fc3e4 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:17.953718 1207 ts_tablet_manager.cc:1428] T 9812754c610548d1b43084bbc94fc3e4 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.378s user 0.006s sys 0.000s
I20250114 02:54:17.954916 1207 tablet_bootstrap.cc:492] T 99b8acb1fe594bf3825c7b9121018db6 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:17.956844 1549 raft_consensus.cc:3054] T 15704785f56e43a3af69f181c75877ce P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:17.962522 1652 raft_consensus.cc:513] T 9812754c610548d1b43084bbc94fc3e4 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:17.964046 1205 tablet_bootstrap.cc:492] T db24bf3db61941ba9b5e0662db23e748 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:17.968470 1652 leader_election.cc:290] T 9812754c610548d1b43084bbc94fc3e4 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Requested vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:17.972536 1205 ts_tablet_manager.cc:1397] T db24bf3db61941ba9b5e0662db23e748 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.502s user 0.009s sys 0.002s
I20250114 02:54:17.972555 1101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9812754c610548d1b43084bbc94fc3e4" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:17.973197 1101 raft_consensus.cc:3054] T 9812754c610548d1b43084bbc94fc3e4 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:17.974833 1205 raft_consensus.cc:357] T db24bf3db61941ba9b5e0662db23e748 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:17.975426 1205 raft_consensus.cc:383] T db24bf3db61941ba9b5e0662db23e748 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:17.975644 1205 raft_consensus.cc:738] T db24bf3db61941ba9b5e0662db23e748 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:17.976204 1205 consensus_queue.cc:260] T db24bf3db61941ba9b5e0662db23e748 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:17.978111 1205 ts_tablet_manager.cc:1428] T db24bf3db61941ba9b5e0662db23e748 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20250114 02:54:17.979911 1205 tablet_bootstrap.cc:492] T 961b6a25a13a4fa2a732a6383dcd8c84 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:17.986267 1205 tablet_bootstrap.cc:654] T 961b6a25a13a4fa2a732a6383dcd8c84 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:17.984623 1594 ts_tablet_manager.cc:1918] T 9589afd0094545c1801b7a0b5eb8698f P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:17.986936 1594 log.cc:1198] T 9589afd0094545c1801b7a0b5eb8698f P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/9589afd0094545c1801b7a0b5eb8698f
I20250114 02:54:17.987690 1594 ts_tablet_manager.cc:1939] T 9589afd0094545c1801b7a0b5eb8698f P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:17.991509 1207 tablet_bootstrap.cc:654] T 99b8acb1fe594bf3825c7b9121018db6 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:17.993165 1595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 115) took 2113 ms. Trace:
I20250114 02:54:17.993489 1595 rpcz_store.cc:276] 0114 02:54:15.879615 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:15.879706 (+ 91us) service_pool.cc:225] Handling call
0114 02:54:17.482647 (+1602941us) ts_tablet_manager.cc:1145] Deleting tablet ed2d137579b8467da5eed4ea2f619a9a
0114 02:54:17.483727 (+ 1080us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:17.536623 (+ 52896us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:17.792195 (+255572us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:17.993104 (+200909us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1592654}
I20250114 02:54:17.994704 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet ed2d137579b8467da5eed4ea2f619a9a (table default.Table_4c3843233dda407395ab9652d5870bc6 [id=0d173fdb502f482fb86b6d3e5fdc66f1]) successfully deleted
I20250114 02:54:17.995541 1595 tablet_replica.cc:331] T 9589afd0094545c1801b7a0b5eb8698f P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:18.001092 1673 raft_consensus.cc:491] T 00080bea6c8e46e584ccba6dcb0e221b P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:18.001506 1673 raft_consensus.cc:513] T 00080bea6c8e46e584ccba6dcb0e221b P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:18.003186 1673 leader_election.cc:290] T 00080bea6c8e46e584ccba6dcb0e221b P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:18.004717 1102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00080bea6c8e46e584ccba6dcb0e221b" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:18.008083 1206 tablet_bootstrap.cc:492] T 99b8acb1fe594bf3825c7b9121018db6 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:18.008584 1206 ts_tablet_manager.cc:1397] T 99b8acb1fe594bf3825c7b9121018db6 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.134s user 0.011s sys 0.004s
I20250114 02:54:18.012648 1205 tablet_bootstrap.cc:492] T 961b6a25a13a4fa2a732a6383dcd8c84 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:18.013016 1205 ts_tablet_manager.cc:1397] T 961b6a25a13a4fa2a732a6383dcd8c84 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.034s user 0.007s sys 0.005s
I20250114 02:54:18.014364 815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00080bea6c8e46e584ccba6dcb0e221b" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:18.015172 815 raft_consensus.cc:2388] T 00080bea6c8e46e584ccba6dcb0e221b P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 525198593c8e47afaee54e74267b87e3 in current term 1: Already voted for candidate 90d6c2c58bb54e0eafb365625ac33871 in this term.
I20250114 02:54:18.014827 467 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_8be12474c1844678b8d3847682d60ee5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:18.021039 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9812754c610548d1b43084bbc94fc3e4" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
I20250114 02:54:18.021811 1205 raft_consensus.cc:357] T 961b6a25a13a4fa2a732a6383dcd8c84 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:18.022433 1205 raft_consensus.cc:383] T 961b6a25a13a4fa2a732a6383dcd8c84 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:18.022660 1205 raft_consensus.cc:738] T 961b6a25a13a4fa2a732a6383dcd8c84 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:18.023272 1205 consensus_queue.cc:260] T 961b6a25a13a4fa2a732a6383dcd8c84 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:18.030047 1205 ts_tablet_manager.cc:1428] T 961b6a25a13a4fa2a732a6383dcd8c84 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.017s user 0.005s sys 0.001s
I20250114 02:54:18.031495 1205 tablet_bootstrap.cc:492] T 9812754c610548d1b43084bbc94fc3e4 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:18.033533 1206 raft_consensus.cc:357] T 99b8acb1fe594bf3825c7b9121018db6 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:18.034240 1206 raft_consensus.cc:383] T 99b8acb1fe594bf3825c7b9121018db6 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:18.036554 1205 tablet_bootstrap.cc:654] T 9812754c610548d1b43084bbc94fc3e4 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:18.045907 898 leader_election.cc:304] T 00080bea6c8e46e584ccba6dcb0e221b P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 525198593c8e47afaee54e74267b87e3; no voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:18.047298 1673 raft_consensus.cc:3054] T 00080bea6c8e46e584ccba6dcb0e221b P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:18.052400 1205 tablet_bootstrap.cc:492] T 9812754c610548d1b43084bbc94fc3e4 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:18.053340 1205 ts_tablet_manager.cc:1397] T 9812754c610548d1b43084bbc94fc3e4 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.000s
I20250114 02:54:18.054450 1673 raft_consensus.cc:2743] T 00080bea6c8e46e584ccba6dcb0e221b P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250114 02:54:18.055416 467 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8be12474c1844678b8d3847682d60ee5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:18.055492 1205 raft_consensus.cc:357] T 9812754c610548d1b43084bbc94fc3e4 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:18.056080 1205 raft_consensus.cc:383] T 9812754c610548d1b43084bbc94fc3e4 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:18.056339 1205 raft_consensus.cc:738] T 9812754c610548d1b43084bbc94fc3e4 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:18.059010 1205 consensus_queue.cc:260] T 9812754c610548d1b43084bbc94fc3e4 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:18.063413 1205 ts_tablet_manager.cc:1428] T 9812754c610548d1b43084bbc94fc3e4 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20250114 02:54:18.064164 1205 tablet_bootstrap.cc:492] T f464eabfbc0945fa8a5583f00c1dab40 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:18.068318 469 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_8344ae9b78e64a038ba7e64f1f807946" } new_table_name: "default.Table_9676926a50eb42349a2d609b30566e9a" modify_external_catalogs: true
I20250114 02:54:18.078717 1205 tablet_bootstrap.cc:654] T f464eabfbc0945fa8a5583f00c1dab40 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:18.081434 1673 consensus_queue.cc:1035] T de4ca83a69704a368a25954cdc1b9392 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:18.034860 1206 raft_consensus.cc:738] T 99b8acb1fe594bf3825c7b9121018db6 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:18.093344 1206 consensus_queue.cc:260] T 99b8acb1fe594bf3825c7b9121018db6 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
W20250114 02:54:18.045351 751 leader_election.cc:343] T 9812754c610548d1b43084bbc94fc3e4 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
W20250114 02:54:18.095229 751 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.66:44037, user_credentials={real_user=slave}} blocked reactor thread for 49914.9us
I20250114 02:54:18.038568 1294 raft_consensus.cc:491] T fcb58a05f980460884ed81a30a5d59c2 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:18.097750 1294 raft_consensus.cc:513] T fcb58a05f980460884ed81a30a5d59c2 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:18.100994 1099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb58a05f980460884ed81a30a5d59c2" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
W20250114 02:54:18.102146 753 leader_election.cc:343] T fcb58a05f980460884ed81a30a5d59c2 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:18.102586 1294 leader_election.cc:290] T fcb58a05f980460884ed81a30a5d59c2 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:18.038465 1549 raft_consensus.cc:2743] T 15704785f56e43a3af69f181c75877ce P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:18.106765 1618 consensus_queue.cc:1035] T 15704785f56e43a3af69f181c75877ce P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:18.109195 818 raft_consensus.cc:3054] T de4ca83a69704a368a25954cdc1b9392 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:18.114468 961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb58a05f980460884ed81a30a5d59c2" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
W20250114 02:54:18.116291 751 leader_election.cc:343] T fcb58a05f980460884ed81a30a5d59c2 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:18.116667 751 leader_election.cc:304] T fcb58a05f980460884ed81a30a5d59c2 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:18.117995 1505 consensus_queue.cc:1035] T 00080bea6c8e46e584ccba6dcb0e221b P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:18.123358 1549 raft_consensus.cc:2743] T fcb58a05f980460884ed81a30a5d59c2 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:18.139657 1206 ts_tablet_manager.cc:1428] T 99b8acb1fe594bf3825c7b9121018db6 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.131s user 0.004s sys 0.001s
I20250114 02:54:18.141402 1101 raft_consensus.cc:2463] T 9812754c610548d1b43084bbc94fc3e4 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 1.
I20250114 02:54:18.142345 753 leader_election.cc:304] T 9812754c610548d1b43084bbc94fc3e4 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters: 525198593c8e47afaee54e74267b87e3
I20250114 02:54:18.143253 1549 raft_consensus.cc:2798] T 9812754c610548d1b43084bbc94fc3e4 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:18.157390 1673 consensus_queue.cc:1035] T 15704785f56e43a3af69f181c75877ce P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250114 02:54:18.164175 464 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1381e26725174b9c877075cf5444f566 consensus state has no leader
I20250114 02:54:18.165098 464 master_service.cc:946] ReplaceTablet: received request to replace tablet bc46815b4803418c95ea9c039a4fae88 from {username='slave'} at 127.0.0.1:33540
I20250114 02:54:18.167076 1549 raft_consensus.cc:695] T 9812754c610548d1b43084bbc94fc3e4 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 LEADER]: Becoming Leader. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Running, Role: LEADER
I20250114 02:54:18.168283 1549 consensus_queue.cc:237] T 9812754c610548d1b43084bbc94fc3e4 P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:18.171496 1505 consensus_queue.cc:1035] T 00080bea6c8e46e584ccba6dcb0e221b P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:18.175834 1618 consensus_queue.cc:1035] T de4ca83a69704a368a25954cdc1b9392 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:54:17.996342 1595 raft_consensus.cc:2238] T 9589afd0094545c1801b7a0b5eb8698f P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:18.188586 1595 raft_consensus.cc:2267] T 9589afd0094545c1801b7a0b5eb8698f P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:18.190464 1595 ts_tablet_manager.cc:1905] T 9589afd0094545c1801b7a0b5eb8698f P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:18.198311 1594 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 102) took 2357 ms. Trace:
I20250114 02:54:18.198647 1594 rpcz_store.cc:276] 0114 02:54:15.884275 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:15.884401 (+ 126us) service_pool.cc:225] Handling call
0114 02:54:17.057358 (+1172957us) ts_tablet_manager.cc:1145] Deleting tablet 9589afd0094545c1801b7a0b5eb8698f
0114 02:54:17.076882 (+ 19524us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:17.741972 (+665090us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:17.984398 (+242426us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:18.198239 (+213841us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1172397}
I20250114 02:54:18.199831 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet 9589afd0094545c1801b7a0b5eb8698f (table default.Table_ed523e66ce6945a7a0fcdb96e498131d [id=aad0de2f82d744ce9054dc12a6b55536]) successfully deleted
I20250114 02:54:18.200528 1206 tablet_bootstrap.cc:492] T f464eabfbc0945fa8a5583f00c1dab40 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:18.203940 1594 tablet_replica.cc:331] T 276942e5e54246c79f0d4f08297f58ea P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:18.204770 1594 raft_consensus.cc:2238] T 276942e5e54246c79f0d4f08297f58ea P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:18.205224 1594 raft_consensus.cc:2267] T 276942e5e54246c79f0d4f08297f58ea P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:18.215222 1594 ts_tablet_manager.cc:1905] T 276942e5e54246c79f0d4f08297f58ea P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:18.223803 1206 tablet_bootstrap.cc:654] T f464eabfbc0945fa8a5583f00c1dab40 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
W20250114 02:54:18.224125 469 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet aad96f2507974023aaa988e6950feb7a consensus state has no leader
I20250114 02:54:18.250057 1205 tablet_bootstrap.cc:492] T f464eabfbc0945fa8a5583f00c1dab40 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:18.250450 1205 ts_tablet_manager.cc:1397] T f464eabfbc0945fa8a5583f00c1dab40 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.186s user 0.026s sys 0.033s
I20250114 02:54:18.252367 1207 tablet_bootstrap.cc:492] T 99b8acb1fe594bf3825c7b9121018db6 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:18.254704 1205 raft_consensus.cc:357] T f464eabfbc0945fa8a5583f00c1dab40 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:18.255339 1205 raft_consensus.cc:383] T f464eabfbc0945fa8a5583f00c1dab40 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:18.255565 1205 raft_consensus.cc:738] T f464eabfbc0945fa8a5583f00c1dab40 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:18.256449 1205 consensus_queue.cc:260] T f464eabfbc0945fa8a5583f00c1dab40 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:18.259264 1207 ts_tablet_manager.cc:1397] T 99b8acb1fe594bf3825c7b9121018db6 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.304s user 0.008s sys 0.001s
I20250114 02:54:18.267796 467 master_service.cc:946] ReplaceTablet: received request to replace tablet a3886c5f976445cebc95ef1839a74aa9 from {username='slave'} at 127.0.0.1:33540
I20250114 02:54:18.281153 1507 consensus_queue.cc:1035] T db24bf3db61941ba9b5e0662db23e748 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:18.286881 1205 ts_tablet_manager.cc:1428] T f464eabfbc0945fa8a5583f00c1dab40 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.036s user 0.006s sys 0.000s
I20250114 02:54:18.263005 1207 raft_consensus.cc:357] T 99b8acb1fe594bf3825c7b9121018db6 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:18.299508 1207 raft_consensus.cc:383] T 99b8acb1fe594bf3825c7b9121018db6 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:18.300312 1207 raft_consensus.cc:738] T 99b8acb1fe594bf3825c7b9121018db6 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:18.302847 1594 ts_tablet_manager.cc:1918] T 276942e5e54246c79f0d4f08297f58ea P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:18.303277 1594 log.cc:1198] T 276942e5e54246c79f0d4f08297f58ea P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/276942e5e54246c79f0d4f08297f58ea
I20250114 02:54:18.304023 1594 ts_tablet_manager.cc:1939] T 276942e5e54246c79f0d4f08297f58ea P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:18.304392 1207 consensus_queue.cc:260] T 99b8acb1fe594bf3825c7b9121018db6 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:18.306146 1594 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 103) took 2420 ms. Trace:
I20250114 02:54:18.306488 1594 rpcz_store.cc:276] 0114 02:54:15.885379 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:15.885470 (+ 91us) service_pool.cc:225] Handling call
0114 02:54:18.203787 (+2318317us) ts_tablet_manager.cc:1145] Deleting tablet 276942e5e54246c79f0d4f08297f58ea
0114 02:54:18.203830 (+ 43us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:18.262166 (+ 58336us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:18.302635 (+ 40469us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:18.306082 (+ 3447us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2317819}
I20250114 02:54:18.307713 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet 276942e5e54246c79f0d4f08297f58ea (table default.Table_b6330aba5846413c915c50dc4c6e5f11 [id=c3d72f049d9849ab86e252c47e5d24e2]) successfully deleted
I20250114 02:54:18.308946 1205 tablet_bootstrap.cc:492] T 99b8acb1fe594bf3825c7b9121018db6 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:18.316723 1594 tablet_replica.cc:331] T db24bf3db61941ba9b5e0662db23e748 P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:18.317553 1594 raft_consensus.cc:2238] T db24bf3db61941ba9b5e0662db23e748 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:18.317950 1594 pending_rounds.cc:62] T db24bf3db61941ba9b5e0662db23e748 P 819f9e3412e54d44abb11d8b3ca877c2: Trying to abort 1 pending ops.
I20250114 02:54:18.318148 1594 pending_rounds.cc:69] T db24bf3db61941ba9b5e0662db23e748 P 819f9e3412e54d44abb11d8b3ca877c2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114028062356017152 op_type: NO_OP noop_request { }
I20250114 02:54:18.318492 1594 raft_consensus.cc:2883] T db24bf3db61941ba9b5e0662db23e748 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250114 02:54:18.318684 795 tablet_service.cc:1467] Processing CreateTablet for tablet 46360f726fad4534a2daf67581346fc1 (DEFAULT_TABLE table=default.Table_3e7cad29b0c0447ab9539b45ee482232 [id=f65443bbd53c47e184a6d2e9e99dd1b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:18.319784 795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46360f726fad4534a2daf67581346fc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:18.320470 1083 tablet_service.cc:1467] Processing CreateTablet for tablet 46360f726fad4534a2daf67581346fc1 (DEFAULT_TABLE table=default.Table_3e7cad29b0c0447ab9539b45ee482232 [id=f65443bbd53c47e184a6d2e9e99dd1b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:18.321417 1083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46360f726fad4534a2daf67581346fc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:18.318817 1594 raft_consensus.cc:2267] T db24bf3db61941ba9b5e0662db23e748 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:18.324174 463 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet 179248cb5acf4740b268e0ec8b2852f9 (table default.Table_58e014a2cfee480184ff3c66b1768cf6 [id=5745501ecd7b4cecaca3d35502a6cc40]). Expected version 2 got 0
I20250114 02:54:18.326484 1594 ts_tablet_manager.cc:1905] T db24bf3db61941ba9b5e0662db23e748 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250114 02:54:18.329340 1042 consensus_peers.cc:487] T db24bf3db61941ba9b5e0662db23e748 P 90d6c2c58bb54e0eafb365625ac33871 -> Peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Couldn't send request to peer 819f9e3412e54d44abb11d8b3ca877c2. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250114 02:54:18.330009 463 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet a3fdab330d574c0283ac58bc9e450b87 (table default.Table_ec0eefb64458440ebd57a05a7ad0bb9a [id=b6874c22fe0d4151a7c8ea061f685938]). Expected version 1 got 0
I20250114 02:54:18.331403 463 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet 4bb9b429af92446bbe13894d5dfe4c47 (table default.Table_acb5bf9354be440bbf6034cf1c535dab [id=22b7edcf4b7e4c31b3c23dc44ccf028e]). Expected version 1 got 0
I20250114 02:54:18.332603 938 tablet_service.cc:1467] Processing CreateTablet for tablet 46360f726fad4534a2daf67581346fc1 (DEFAULT_TABLE table=default.Table_3e7cad29b0c0447ab9539b45ee482232 [id=f65443bbd53c47e184a6d2e9e99dd1b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:18.333629 938 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46360f726fad4534a2daf67581346fc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:18.338369 961 raft_consensus.cc:3054] T db24bf3db61941ba9b5e0662db23e748 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:18.340257 1205 tablet_bootstrap.cc:654] T 99b8acb1fe594bf3825c7b9121018db6 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:18.348727 1594 ts_tablet_manager.cc:1918] T db24bf3db61941ba9b5e0662db23e748 P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:18.349092 1594 log.cc:1198] T db24bf3db61941ba9b5e0662db23e748 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/db24bf3db61941ba9b5e0662db23e748
I20250114 02:54:18.349092 1507 consensus_queue.cc:1035] T db24bf3db61941ba9b5e0662db23e748 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:18.349783 1594 ts_tablet_manager.cc:1939] T db24bf3db61941ba9b5e0662db23e748 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:18.361277 1206 tablet_bootstrap.cc:492] T f464eabfbc0945fa8a5583f00c1dab40 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:18.361698 1206 ts_tablet_manager.cc:1397] T f464eabfbc0945fa8a5583f00c1dab40 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.161s user 0.012s sys 0.000s
I20250114 02:54:18.361788 1205 tablet_bootstrap.cc:492] T 99b8acb1fe594bf3825c7b9121018db6 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:18.361668 463 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet de4ca83a69704a368a25954cdc1b9392 (table default.Table_3139140b7eec4849ae715c295fbeaa6b [id=8536eb7f6bf542daa88a8db0e4873456]). Expected version 2 got 0
I20250114 02:54:18.362165 1205 ts_tablet_manager.cc:1397] T 99b8acb1fe594bf3825c7b9121018db6 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.053s user 0.007s sys 0.003s
I20250114 02:54:18.362793 463 catalog_manager.cc:5526] T e3af8f1299f24b08b2d25811942aaecc P 90d6c2c58bb54e0eafb365625ac33871 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67). New cstate: current_term: 1 leader_uuid: "90d6c2c58bb54e0eafb365625ac33871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } }
I20250114 02:54:18.363762 1206 raft_consensus.cc:357] T f464eabfbc0945fa8a5583f00c1dab40 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:18.364416 1206 raft_consensus.cc:383] T f464eabfbc0945fa8a5583f00c1dab40 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:18.364642 1206 raft_consensus.cc:738] T f464eabfbc0945fa8a5583f00c1dab40 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:18.365211 1206 consensus_queue.cc:260] T f464eabfbc0945fa8a5583f00c1dab40 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:18.365981 467 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet a3886c5f976445cebc95ef1839a74aa9
I20250114 02:54:18.367416 1080 tablet_service.cc:1514] Processing DeleteTablet for tablet a3886c5f976445cebc95ef1839a74aa9 with delete_type TABLET_DATA_DELETED (replaced by tablet 3161f75bc4e84000af022faa9804c810) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:18.368160 467 catalog_manager.cc:6596] ReplaceTablet: tablet a3886c5f976445cebc95ef1839a74aa9 deleted and replaced by tablet 3161f75bc4e84000af022faa9804c810
I20250114 02:54:18.368736 937 tablet_service.cc:1514] Processing DeleteTablet for tablet a3886c5f976445cebc95ef1839a74aa9 with delete_type TABLET_DATA_DELETED (replaced by tablet 3161f75bc4e84000af022faa9804c810) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:18.368722 798 tablet_service.cc:1514] Processing DeleteTablet for tablet a3886c5f976445cebc95ef1839a74aa9 with delete_type TABLET_DATA_DELETED (replaced by tablet 3161f75bc4e84000af022faa9804c810) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:18.367154 1206 ts_tablet_manager.cc:1428] T f464eabfbc0945fa8a5583f00c1dab40 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20250114 02:54:18.369951 1206 tablet_bootstrap.cc:492] T aad96f2507974023aaa988e6950feb7a P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:18.375845 467 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_693a2c1631b347af8d9d873d6b08eb92"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:18.377430 467 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_693a2c1631b347af8d9d873d6b08eb92 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:18.381132 1206 tablet_bootstrap.cc:654] T aad96f2507974023aaa988e6950feb7a P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:18.381744 1207 ts_tablet_manager.cc:1428] T 99b8acb1fe594bf3825c7b9121018db6 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.121s user 0.003s sys 0.002s
I20250114 02:54:18.382668 1207 tablet_bootstrap.cc:492] T fcb58a05f980460884ed81a30a5d59c2 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:18.383031 1702 tablet_replica.cc:331] T a3886c5f976445cebc95ef1839a74aa9 P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:18.383841 1702 raft_consensus.cc:2238] T a3886c5f976445cebc95ef1839a74aa9 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:18.386445 1702 raft_consensus.cc:2267] T a3886c5f976445cebc95ef1839a74aa9 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:18.390244 798 tablet_service.cc:1467] Processing CreateTablet for tablet f0bd9bd5a17d418d8082ed0ebd1bf4cc (DEFAULT_TABLE table=default.Table_dc087fb027f74c4a8bd5dede1142d97d [id=971cb3ca793e4945ad07e6be966e01f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:18.391216 798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0bd9bd5a17d418d8082ed0ebd1bf4cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:18.364178 1205 raft_consensus.cc:357] T 99b8acb1fe594bf3825c7b9121018db6 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:18.385887 1594 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 104) took 2463 ms. Trace:
I20250114 02:54:18.391772 1205 raft_consensus.cc:383] T 99b8acb1fe594bf3825c7b9121018db6 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:18.391996 1205 raft_consensus.cc:738] T 99b8acb1fe594bf3825c7b9121018db6 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:18.391950 1594 rpcz_store.cc:276] 0114 02:54:15.922272 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:15.922404 (+ 132us) service_pool.cc:225] Handling call
0114 02:54:18.316564 (+2394160us) ts_tablet_manager.cc:1145] Deleting tablet db24bf3db61941ba9b5e0662db23e748
0114 02:54:18.316609 (+ 45us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:18.342132 (+ 25523us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:18.348507 (+ 6375us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:18.385827 (+ 37320us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2393498}
I20250114 02:54:18.392645 1205 consensus_queue.cc:260] T 99b8acb1fe594bf3825c7b9121018db6 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:18.393193 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet db24bf3db61941ba9b5e0662db23e748 (table default.Table_494541d472dc48d0bef33a634c1180fa [id=5ae72503278349c0bbe315174c896074]) successfully deleted
I20250114 02:54:18.394034 1594 tablet_replica.cc:331] T 96b06fe4173343dabd726d253b9f9b5f P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:18.394774 1594 raft_consensus.cc:2238] T 96b06fe4173343dabd726d253b9f9b5f P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:18.395170 1594 raft_consensus.cc:2267] T 96b06fe4173343dabd726d253b9f9b5f P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:18.395745 1205 ts_tablet_manager.cc:1428] T 99b8acb1fe594bf3825c7b9121018db6 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.033s user 0.005s sys 0.001s
I20250114 02:54:18.396514 1205 tablet_bootstrap.cc:492] T fcb58a05f980460884ed81a30a5d59c2 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:18.396968 1594 ts_tablet_manager.cc:1905] T 96b06fe4173343dabd726d253b9f9b5f P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250114 02:54:18.398192 751 consensus_peers.cc:487] T a3886c5f976445cebc95ef1839a74aa9 P 819f9e3412e54d44abb11d8b3ca877c2 -> Peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Couldn't send request to peer 525198593c8e47afaee54e74267b87e3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250114 02:54:18.388398 1702 ts_tablet_manager.cc:1905] T a3886c5f976445cebc95ef1839a74aa9 P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:18.401141 937 tablet_service.cc:1467] Processing CreateTablet for tablet f0bd9bd5a17d418d8082ed0ebd1bf4cc (DEFAULT_TABLE table=default.Table_dc087fb027f74c4a8bd5dede1142d97d [id=971cb3ca793e4945ad07e6be966e01f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:18.402042 937 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0bd9bd5a17d418d8082ed0ebd1bf4cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:18.427963 1594 ts_tablet_manager.cc:1918] T 96b06fe4173343dabd726d253b9f9b5f P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:18.429767 1207 tablet_bootstrap.cc:654] T fcb58a05f980460884ed81a30a5d59c2 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:18.431553 1206 tablet_bootstrap.cc:492] T aad96f2507974023aaa988e6950feb7a P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:18.432008 1206 ts_tablet_manager.cc:1397] T aad96f2507974023aaa988e6950feb7a P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.062s user 0.011s sys 0.011s
I20250114 02:54:18.434355 1595 ts_tablet_manager.cc:1918] T 9589afd0094545c1801b7a0b5eb8698f P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:18.434752 1595 log.cc:1198] T 9589afd0094545c1801b7a0b5eb8698f P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/9589afd0094545c1801b7a0b5eb8698f
I20250114 02:54:18.434705 1206 raft_consensus.cc:357] T aad96f2507974023aaa988e6950feb7a P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:18.435456 1595 ts_tablet_manager.cc:1939] T 9589afd0094545c1801b7a0b5eb8698f P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
I20250114 02:54:18.435508 1206 raft_consensus.cc:383] T aad96f2507974023aaa988e6950feb7a P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:18.435825 1206 raft_consensus.cc:738] T aad96f2507974023aaa988e6950feb7a P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:18.436592 1206 consensus_queue.cc:260] T aad96f2507974023aaa988e6950feb7a P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:18.438640 1206 ts_tablet_manager.cc:1428] T aad96f2507974023aaa988e6950feb7a P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250114 02:54:18.439453 1206 tablet_bootstrap.cc:492] T ba215eb8422d4f879dd0bf5996543962 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:18.444797 1206 tablet_bootstrap.cc:654] T ba215eb8422d4f879dd0bf5996543962 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:18.428591 1594 log.cc:1198] T 96b06fe4173343dabd726d253b9f9b5f P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/96b06fe4173343dabd726d253b9f9b5f
I20250114 02:54:18.445061 465 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet 53a8bb487ad64157ad3baecf5d866eca
I20250114 02:54:18.445519 1594 ts_tablet_manager.cc:1939] T 96b06fe4173343dabd726d253b9f9b5f P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:18.448529 1080 tablet_service.cc:1514] Processing DeleteTablet for tablet 53a8bb487ad64157ad3baecf5d866eca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:17 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:18.450734 795 tablet_service.cc:1514] Processing DeleteTablet for tablet 53a8bb487ad64157ad3baecf5d866eca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:17 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:18.453204 941 tablet_service.cc:1514] Processing DeleteTablet for tablet 53a8bb487ad64157ad3baecf5d866eca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:17 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:18.455713 1594 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 105) took 2419 ms. Trace:
I20250114 02:54:18.456022 1594 rpcz_store.cc:276] 0114 02:54:16.035852 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:16.035988 (+ 136us) service_pool.cc:225] Handling call
0114 02:54:18.393890 (+2357902us) ts_tablet_manager.cc:1145] Deleting tablet 96b06fe4173343dabd726d253b9f9b5f
0114 02:54:18.393930 (+ 40us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:18.411001 (+ 17071us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:18.427714 (+ 16713us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:18.455656 (+ 27942us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2357276}
I20250114 02:54:18.457011 1594 tablet_replica.cc:331] T b63dd90f60a04465b948171e74888e79 P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:18.457659 1594 raft_consensus.cc:2238] T b63dd90f60a04465b948171e74888e79 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:18.458071 1594 raft_consensus.cc:2267] T b63dd90f60a04465b948171e74888e79 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:18.459618 1594 ts_tablet_manager.cc:1905] T b63dd90f60a04465b948171e74888e79 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:18.460611 1086 tablet_service.cc:1467] Processing CreateTablet for tablet 42cee9e8e26147b593ed73fca6ac1183 (DEFAULT_TABLE table=default.Table_6bdcd457ba84407e8ec4b0fb6a1d04c0 [id=b0cedff1a9b44e249739d2dc53418a16]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:18.461491 1086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42cee9e8e26147b593ed73fca6ac1183. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:18.465180 1080 tablet_service.cc:1467] Processing CreateTablet for tablet f0bd9bd5a17d418d8082ed0ebd1bf4cc (DEFAULT_TABLE table=default.Table_dc087fb027f74c4a8bd5dede1142d97d [id=971cb3ca793e4945ad07e6be966e01f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:18.466146 1080 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0bd9bd5a17d418d8082ed0ebd1bf4cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:18.471534 941 tablet_service.cc:1467] Processing CreateTablet for tablet 42cee9e8e26147b593ed73fca6ac1183 (DEFAULT_TABLE table=default.Table_6bdcd457ba84407e8ec4b0fb6a1d04c0 [id=b0cedff1a9b44e249739d2dc53418a16]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:18.472472 941 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42cee9e8e26147b593ed73fca6ac1183. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:18.475984 794 tablet_service.cc:1467] Processing CreateTablet for tablet 84738a6c78b24c1fb64013048763b18f (DEFAULT_TABLE table=default.Table_1c87461bd6f64248aecf8fd2d9d3700b [id=b5d3461f83da46faaa772376a318b6b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:18.475895 795 tablet_service.cc:1467] Processing CreateTablet for tablet 42cee9e8e26147b593ed73fca6ac1183 (DEFAULT_TABLE table=default.Table_6bdcd457ba84407e8ec4b0fb6a1d04c0 [id=b0cedff1a9b44e249739d2dc53418a16]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:18.476910 794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84738a6c78b24c1fb64013048763b18f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:18.477438 795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42cee9e8e26147b593ed73fca6ac1183. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:18.477530 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet 96b06fe4173343dabd726d253b9f9b5f (table default.Table_d1de82caa5b94791b6c7dfed1c376e8a [id=03c775a54e9e443cbc4fab853bd6d49c]) successfully deleted
I20250114 02:54:18.452348 1673 raft_consensus.cc:491] T f464eabfbc0945fa8a5583f00c1dab40 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:18.478063 1673 raft_consensus.cc:513] T f464eabfbc0945fa8a5583f00c1dab40 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:18.479902 1673 leader_election.cc:290] T f464eabfbc0945fa8a5583f00c1dab40 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:18.490478 1207 tablet_bootstrap.cc:492] T fcb58a05f980460884ed81a30a5d59c2 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:18.495887 465 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_fee886c2440a47bfa5a5875d58ef45ad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:18.496964 1207 ts_tablet_manager.cc:1397] T fcb58a05f980460884ed81a30a5d59c2 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.114s user 0.009s sys 0.001s
W20250114 02:54:18.497428 465 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fee886c2440a47bfa5a5875d58ef45ad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:18.499344 1207 raft_consensus.cc:357] T fcb58a05f980460884ed81a30a5d59c2 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:18.500044 1207 raft_consensus.cc:383] T fcb58a05f980460884ed81a30a5d59c2 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:18.500377 1207 raft_consensus.cc:738] T fcb58a05f980460884ed81a30a5d59c2 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:18.496454 1595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 116) took 2614 ms. Trace:
I20250114 02:54:18.500981 1595 rpcz_store.cc:276] 0114 02:54:15.881803 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:15.881992 (+ 189us) service_pool.cc:225] Handling call
0114 02:54:17.995394 (+2113402us) ts_tablet_manager.cc:1145] Deleting tablet 9589afd0094545c1801b7a0b5eb8698f
0114 02:54:17.995444 (+ 50us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:18.365852 (+370408us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:18.434125 (+ 68273us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:18.496388 (+ 62263us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2112741}
I20250114 02:54:18.503299 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet 9589afd0094545c1801b7a0b5eb8698f (table default.Table_ed523e66ce6945a7a0fcdb96e498131d [id=aad0de2f82d744ce9054dc12a6b55536]) successfully deleted
I20250114 02:54:18.509660 1101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f464eabfbc0945fa8a5583f00c1dab40" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:18.510962 816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f464eabfbc0945fa8a5583f00c1dab40" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:18.511575 816 raft_consensus.cc:2463] T f464eabfbc0945fa8a5583f00c1dab40 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:18.512718 898 leader_election.cc:304] T f464eabfbc0945fa8a5583f00c1dab40 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
W20250114 02:54:18.513640 898 leader_election.cc:343] T f464eabfbc0945fa8a5583f00c1dab40 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:18.517244 792 tablet_service.cc:1467] Processing CreateTablet for tablet ce7d6088fc884a248918d2348adcaccf (DEFAULT_TABLE table=default.Table_11c84abe27d74b67bfbd28c249c78b03 [id=35c5879da4b344118a377335fc6f83de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:18.518151 792 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce7d6088fc884a248918d2348adcaccf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:18.519526 1083 tablet_service.cc:1467] Processing CreateTablet for tablet 84738a6c78b24c1fb64013048763b18f (DEFAULT_TABLE table=default.Table_1c87461bd6f64248aecf8fd2d9d3700b [id=b5d3461f83da46faaa772376a318b6b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:18.520514 1083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84738a6c78b24c1fb64013048763b18f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:18.522753 1206 tablet_bootstrap.cc:492] T ba215eb8422d4f879dd0bf5996543962 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:18.523176 1206 ts_tablet_manager.cc:1397] T ba215eb8422d4f879dd0bf5996543962 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.084s user 0.013s sys 0.001s
I20250114 02:54:18.524583 1081 tablet_service.cc:1467] Processing CreateTablet for tablet ce7d6088fc884a248918d2348adcaccf (DEFAULT_TABLE table=default.Table_11c84abe27d74b67bfbd28c249c78b03 [id=35c5879da4b344118a377335fc6f83de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:18.524994 791 tablet_service.cc:1467] Processing CreateTablet for tablet 5ac7708550954a209e2a8d217c3b39bd (DEFAULT_TABLE table=default.Table_d50b0a8b26f644898efcb62a0d72fd1b [id=13151866bf2b4f44b1dd5c92380a4d43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:18.525502 1081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce7d6088fc884a248918d2348adcaccf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:18.525991 791 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ac7708550954a209e2a8d217c3b39bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:18.534235 936 tablet_service.cc:1467] Processing CreateTablet for tablet ce7d6088fc884a248918d2348adcaccf (DEFAULT_TABLE table=default.Table_11c84abe27d74b67bfbd28c249c78b03 [id=35c5879da4b344118a377335fc6f83de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:18.536685 940 tablet_service.cc:1467] Processing CreateTablet for tablet 5ac7708550954a209e2a8d217c3b39bd (DEFAULT_TABLE table=default.Table_d50b0a8b26f644898efcb62a0d72fd1b [id=13151866bf2b4f44b1dd5c92380a4d43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:18.532883 1673 raft_consensus.cc:2798] T f464eabfbc0945fa8a5583f00c1dab40 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:18.538205 1673 raft_consensus.cc:491] T f464eabfbc0945fa8a5583f00c1dab40 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:18.538496 1673 raft_consensus.cc:3054] T f464eabfbc0945fa8a5583f00c1dab40 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:18.541615 1206 raft_consensus.cc:357] T ba215eb8422d4f879dd0bf5996543962 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:18.542229 1206 raft_consensus.cc:383] T ba215eb8422d4f879dd0bf5996543962 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:18.542464 1206 raft_consensus.cc:738] T ba215eb8422d4f879dd0bf5996543962 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:18.543042 1206 consensus_queue.cc:260] T ba215eb8422d4f879dd0bf5996543962 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:18.545279 1206 ts_tablet_manager.cc:1428] T ba215eb8422d4f879dd0bf5996543962 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.022s user 0.003s sys 0.001s
I20250114 02:54:18.547003 939 tablet_service.cc:1467] Processing CreateTablet for tablet 84738a6c78b24c1fb64013048763b18f (DEFAULT_TABLE table=default.Table_1c87461bd6f64248aecf8fd2d9d3700b [id=b5d3461f83da46faaa772376a318b6b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:18.553992 1594 ts_tablet_manager.cc:1918] T b63dd90f60a04465b948171e74888e79 P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250114 02:54:18.554476 1594 log.cc:1198] T b63dd90f60a04465b948171e74888e79 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/b63dd90f60a04465b948171e74888e79
I20250114 02:54:18.564635 1595 tablet_replica.cc:331] T c20aca761313498193e1d79c10705f8f P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:18.565510 1595 raft_consensus.cc:2238] T c20aca761313498193e1d79c10705f8f P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Raft consensus shutting down.
I20250114 02:54:18.566458 1595 raft_consensus.cc:2267] T c20aca761313498193e1d79c10705f8f P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:18.572650 1594 ts_tablet_manager.cc:1939] T b63dd90f60a04465b948171e74888e79 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:18.502072 1207 consensus_queue.cc:260] T fcb58a05f980460884ed81a30a5d59c2 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:18.580281 938 tablet_service.cc:1467] Processing CreateTablet for tablet e15e6e52d7294aeeb4463b3a76022ec3 (DEFAULT_TABLE table=default.Table_d1de82caa5b94791b6c7dfed1c376e8a [id=03c775a54e9e443cbc4fab853bd6d49c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:18.580704 942 tablet_service.cc:1467] Processing CreateTablet for tablet ddfbab181b1f474192c664cd0316037f (DEFAULT_TABLE table=default.Table_494541d472dc48d0bef33a634c1180fa [id=5ae72503278349c0bbe315174c896074]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:18.580893 935 tablet_service.cc:1467] Processing CreateTablet for tablet 92cd8eccacf34ab7a66b3bbc69df3697 (DEFAULT_TABLE table=default.Table_fa5f4f55ea844a8c8af8acb80f0fd6a4 [id=26627ccdc808451fb9593ad7a6df6991]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:18.582216 1207 ts_tablet_manager.cc:1428] T fcb58a05f980460884ed81a30a5d59c2 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.085s user 0.004s sys 0.000s
I20250114 02:54:18.586582 1080 tablet_service.cc:1467] Processing CreateTablet for tablet e15e6e52d7294aeeb4463b3a76022ec3 (DEFAULT_TABLE table=default.Table_d1de82caa5b94791b6c7dfed1c376e8a [id=03c775a54e9e443cbc4fab853bd6d49c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:18.587307 1082 tablet_service.cc:1467] Processing CreateTablet for tablet ddfbab181b1f474192c664cd0316037f (DEFAULT_TABLE table=default.Table_494541d472dc48d0bef33a634c1180fa [id=5ae72503278349c0bbe315174c896074]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:18.590107 798 tablet_service.cc:1467] Processing CreateTablet for tablet ddfbab181b1f474192c664cd0316037f (DEFAULT_TABLE table=default.Table_494541d472dc48d0bef33a634c1180fa [id=5ae72503278349c0bbe315174c896074]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:18.591540 798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddfbab181b1f474192c664cd0316037f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:18.593194 1082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddfbab181b1f474192c664cd0316037f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:18.592999 795 tablet_service.cc:1467] Processing CreateTablet for tablet e15e6e52d7294aeeb4463b3a76022ec3 (DEFAULT_TABLE table=default.Table_d1de82caa5b94791b6c7dfed1c376e8a [id=03c775a54e9e443cbc4fab853bd6d49c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:18.593932 795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e15e6e52d7294aeeb4463b3a76022ec3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:18.593956 1205 tablet_bootstrap.cc:654] T fcb58a05f980460884ed81a30a5d59c2 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:18.597340 935 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92cd8eccacf34ab7a66b3bbc69df3697. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:18.600453 1086 tablet_service.cc:1467] Processing CreateTablet for tablet 5ac7708550954a209e2a8d217c3b39bd (DEFAULT_TABLE table=default.Table_d50b0a8b26f644898efcb62a0d72fd1b [id=13151866bf2b4f44b1dd5c92380a4d43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:18.600704 936 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce7d6088fc884a248918d2348adcaccf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:18.601369 1086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ac7708550954a209e2a8d217c3b39bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:18.604629 940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ac7708550954a209e2a8d217c3b39bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:18.591542 1673 raft_consensus.cc:513] T f464eabfbc0945fa8a5583f00c1dab40 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:18.608851 939 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84738a6c78b24c1fb64013048763b18f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:18.611511 1207 tablet_bootstrap.cc:492] T aad96f2507974023aaa988e6950feb7a P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:18.612869 942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddfbab181b1f474192c664cd0316037f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:18.623275 1618 raft_consensus.cc:491] T 99b8acb1fe594bf3825c7b9121018db6 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:18.623704 1618 raft_consensus.cc:513] T 99b8acb1fe594bf3825c7b9121018db6 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:18.625464 1618 leader_election.cc:290] T 99b8acb1fe594bf3825c7b9121018db6 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:18.626291 1080 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e15e6e52d7294aeeb4463b3a76022ec3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:18.626945 1101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99b8acb1fe594bf3825c7b9121018db6" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:18.627514 1101 raft_consensus.cc:2463] T 99b8acb1fe594bf3825c7b9121018db6 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:18.628391 816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99b8acb1fe594bf3825c7b9121018db6" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:18.628962 816 raft_consensus.cc:2463] T 99b8acb1fe594bf3825c7b9121018db6 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:18.629290 938 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e15e6e52d7294aeeb4463b3a76022ec3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:18.630429 898 leader_election.cc:304] T 99b8acb1fe594bf3825c7b9121018db6 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
I20250114 02:54:18.632450 1597 raft_consensus.cc:2798] T 99b8acb1fe594bf3825c7b9121018db6 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:18.632488 1101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f464eabfbc0945fa8a5583f00c1dab40" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:18.632730 1597 raft_consensus.cc:491] T 99b8acb1fe594bf3825c7b9121018db6 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:18.632972 1597 raft_consensus.cc:3054] T 99b8acb1fe594bf3825c7b9121018db6 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:18.633996 816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f464eabfbc0945fa8a5583f00c1dab40" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:18.634472 816 raft_consensus.cc:3054] T f464eabfbc0945fa8a5583f00c1dab40 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
W20250114 02:54:18.640599 898 leader_election.cc:343] T f464eabfbc0945fa8a5583f00c1dab40 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:18.641417 1207 tablet_bootstrap.cc:654] T aad96f2507974023aaa988e6950feb7a P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:18.642097 1597 raft_consensus.cc:513] T 99b8acb1fe594bf3825c7b9121018db6 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:18.645403 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99b8acb1fe594bf3825c7b9121018db6" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:18.645888 818 raft_consensus.cc:3054] T 99b8acb1fe594bf3825c7b9121018db6 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:18.646059 1597 leader_election.cc:290] T 99b8acb1fe594bf3825c7b9121018db6 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Requested vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:18.651270 818 raft_consensus.cc:2463] T 99b8acb1fe594bf3825c7b9121018db6 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:18.652412 1101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99b8acb1fe594bf3825c7b9121018db6" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:18.653039 1595 ts_tablet_manager.cc:1905] T c20aca761313498193e1d79c10705f8f P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:18.653463 1206 tablet_bootstrap.cc:492] T 321b9ad6d1ac4d1c80c7a01faf2015af P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:18.653826 1594 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 112) took 1759 ms. Trace:
I20250114 02:54:18.654114 1594 rpcz_store.cc:276] 0114 02:54:16.894158 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:17.053918 (+159760us) service_pool.cc:225] Handling call
0114 02:54:18.456884 (+1402966us) ts_tablet_manager.cc:1145] Deleting tablet b63dd90f60a04465b948171e74888e79
0114 02:54:18.456920 (+ 36us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:18.503149 (+ 46229us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:18.553786 (+ 50637us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:18.653776 (+ 99990us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":80395,"tablet-delete.queue_time_us":1388371}
I20250114 02:54:18.655166 1594 tablet_replica.cc:331] T 961b6a25a13a4fa2a732a6383dcd8c84 P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:18.655828 1594 raft_consensus.cc:2238] T 961b6a25a13a4fa2a732a6383dcd8c84 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 LEADER]: Raft consensus shutting down.
I20250114 02:54:18.656463 1594 pending_rounds.cc:62] T 961b6a25a13a4fa2a732a6383dcd8c84 P 819f9e3412e54d44abb11d8b3ca877c2: Trying to abort 1 pending ops.
I20250114 02:54:18.656666 1594 pending_rounds.cc:69] T 961b6a25a13a4fa2a732a6383dcd8c84 P 819f9e3412e54d44abb11d8b3ca877c2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114028064454742016 op_type: NO_OP noop_request { }
I20250114 02:54:18.656957 1594 raft_consensus.cc:2883] T 961b6a25a13a4fa2a732a6383dcd8c84 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250114 02:54:18.657246 1594 raft_consensus.cc:2267] T 961b6a25a13a4fa2a732a6383dcd8c84 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:18.659766 1206 tablet_bootstrap.cc:654] T 321b9ad6d1ac4d1c80c7a01faf2015af P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:18.660705 1594 ts_tablet_manager.cc:1905] T 961b6a25a13a4fa2a732a6383dcd8c84 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:18.661700 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet b63dd90f60a04465b948171e74888e79 (table default.Table_1097f09fb10e441fb639f8d7aa12d487 [id=d8c9691818d042d7b49475fa009c4ca1]) successfully deleted
I20250114 02:54:18.653107 1101 raft_consensus.cc:3054] T 99b8acb1fe594bf3825c7b9121018db6 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:18.631150 1673 leader_election.cc:290] T f464eabfbc0945fa8a5583f00c1dab40 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Requested vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:18.668817 898 leader_election.cc:304] T 99b8acb1fe594bf3825c7b9121018db6 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
W20250114 02:54:18.669340 455 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.124:39105, user_credentials={real_user=slave}} blocked reactor thread for 82894us
W20250114 02:54:18.670066 455 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 1381e26725174b9c877075cf5444f566 (table default.Table_b624816662024c55a5f5ef5830b20b7c [id=d2ea84ea59ad4b03ad16bb3052a441f4], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 1381e26725174b9c877075cf5444f566 consensus state has no leader
I20250114 02:54:18.670528 1673 raft_consensus.cc:2798] T 99b8acb1fe594bf3825c7b9121018db6 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:18.670251 816 raft_consensus.cc:2463] T f464eabfbc0945fa8a5583f00c1dab40 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
W20250114 02:54:18.672276 456 catalog_manager.cc:4945] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): alter failed for tablet b63dd90f60a04465b948171e74888e79 (table default.Table_1097f09fb10e441fb639f8d7aa12d487 [id=d8c9691818d042d7b49475fa009c4ca1]),no further retry: Not found: Tablet not found: b63dd90f60a04465b948171e74888e79
I20250114 02:54:18.674255 1207 tablet_bootstrap.cc:492] T aad96f2507974023aaa988e6950feb7a P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:18.674264 1673 raft_consensus.cc:695] T 99b8acb1fe594bf3825c7b9121018db6 P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Becoming Leader. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Running, Role: LEADER
I20250114 02:54:18.674706 1207 ts_tablet_manager.cc:1397] T aad96f2507974023aaa988e6950feb7a P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.063s user 0.008s sys 0.004s
I20250114 02:54:18.675072 1673 consensus_queue.cc:237] T 99b8acb1fe594bf3825c7b9121018db6 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:18.676780 464 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet bc46815b4803418c95ea9c039a4fae88
I20250114 02:54:18.676882 1207 raft_consensus.cc:357] T aad96f2507974023aaa988e6950feb7a P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:18.677508 1207 raft_consensus.cc:383] T aad96f2507974023aaa988e6950feb7a P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:18.677739 1207 raft_consensus.cc:738] T aad96f2507974023aaa988e6950feb7a P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:18.677956 464 catalog_manager.cc:6596] ReplaceTablet: tablet bc46815b4803418c95ea9c039a4fae88 deleted and replaced by tablet 97f4e061d3b94ee9982410b7ce135453
I20250114 02:54:18.678385 1207 consensus_queue.cc:260] T aad96f2507974023aaa988e6950feb7a P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:18.677673 898 leader_election.cc:304] T f464eabfbc0945fa8a5583f00c1dab40 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters: 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:18.678857 1702 ts_tablet_manager.cc:1918] T a3886c5f976445cebc95ef1839a74aa9 P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250114 02:54:18.679627 1702 log.cc:1198] T a3886c5f976445cebc95ef1839a74aa9 P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/a3886c5f976445cebc95ef1839a74aa9
I20250114 02:54:18.679937 1597 raft_consensus.cc:2798] T f464eabfbc0945fa8a5583f00c1dab40 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:18.680577 1597 raft_consensus.cc:695] T f464eabfbc0945fa8a5583f00c1dab40 P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Becoming Leader. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Running, Role: LEADER
I20250114 02:54:18.682101 937 tablet_service.cc:1514] Processing DeleteTablet for tablet bc46815b4803418c95ea9c039a4fae88 with delete_type TABLET_DATA_DELETED (replaced by tablet 97f4e061d3b94ee9982410b7ce135453) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:18.681286 1597 consensus_queue.cc:237] T f464eabfbc0945fa8a5583f00c1dab40 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:18.684428 1101 raft_consensus.cc:2463] T 99b8acb1fe594bf3825c7b9121018db6 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:18.686285 795 tablet_service.cc:1514] Processing DeleteTablet for tablet bc46815b4803418c95ea9c039a4fae88 with delete_type TABLET_DATA_DELETED (replaced by tablet 97f4e061d3b94ee9982410b7ce135453) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:18.691615 1206 tablet_bootstrap.cc:492] T 321b9ad6d1ac4d1c80c7a01faf2015af P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:18.692036 1206 ts_tablet_manager.cc:1397] T 321b9ad6d1ac4d1c80c7a01faf2015af P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.039s user 0.006s sys 0.004s
I20250114 02:54:18.693454 1080 tablet_service.cc:1514] Processing DeleteTablet for tablet bc46815b4803418c95ea9c039a4fae88 with delete_type TABLET_DATA_DELETED (replaced by tablet 97f4e061d3b94ee9982410b7ce135453) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:18.680402 1702 ts_tablet_manager.cc:1939] T a3886c5f976445cebc95ef1839a74aa9 P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:18.695246 1206 raft_consensus.cc:357] T 321b9ad6d1ac4d1c80c7a01faf2015af P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:18.696700 1206 raft_consensus.cc:383] T 321b9ad6d1ac4d1c80c7a01faf2015af P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:18.697010 1206 raft_consensus.cc:738] T 321b9ad6d1ac4d1c80c7a01faf2015af P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:18.697119 1207 ts_tablet_manager.cc:1428] T aad96f2507974023aaa988e6950feb7a P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.022s user 0.005s sys 0.000s
I20250114 02:54:18.697647 1206 consensus_queue.cc:260] T 321b9ad6d1ac4d1c80c7a01faf2015af P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:18.703415 1205 tablet_bootstrap.cc:492] T fcb58a05f980460884ed81a30a5d59c2 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:18.703847 1205 ts_tablet_manager.cc:1397] T fcb58a05f980460884ed81a30a5d59c2 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.307s user 0.003s sys 0.006s
I20250114 02:54:18.706341 1205 raft_consensus.cc:357] T fcb58a05f980460884ed81a30a5d59c2 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:18.706976 1205 raft_consensus.cc:383] T fcb58a05f980460884ed81a30a5d59c2 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:18.707249 1205 raft_consensus.cc:738] T fcb58a05f980460884ed81a30a5d59c2 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:18.709669 1207 tablet_bootstrap.cc:492] T f464eabfbc0945fa8a5583f00c1dab40 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:18.712507 795 tablet_service.cc:1467] Processing CreateTablet for tablet 92cd8eccacf34ab7a66b3bbc69df3697 (DEFAULT_TABLE table=default.Table_fa5f4f55ea844a8c8af8acb80f0fd6a4 [id=26627ccdc808451fb9593ad7a6df6991]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:18.713534 795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92cd8eccacf34ab7a66b3bbc69df3697. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:18.713259 1080 tablet_service.cc:1467] Processing CreateTablet for tablet 92cd8eccacf34ab7a66b3bbc69df3697 (DEFAULT_TABLE table=default.Table_fa5f4f55ea844a8c8af8acb80f0fd6a4 [id=26627ccdc808451fb9593ad7a6df6991]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:18.714224 1080 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92cd8eccacf34ab7a66b3bbc69df3697. 1 dirs total, 0 dirs full, 0 dirs failed
W20250114 02:54:18.718703 455 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.125:46837, user_credentials={real_user=slave}} blocked reactor thread for 45495.1us
I20250114 02:54:18.723477 1594 ts_tablet_manager.cc:1918] T 961b6a25a13a4fa2a732a6383dcd8c84 P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:18.723824 1594 log.cc:1198] T 961b6a25a13a4fa2a732a6383dcd8c84 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/961b6a25a13a4fa2a732a6383dcd8c84
I20250114 02:54:18.724576 1594 ts_tablet_manager.cc:1939] T 961b6a25a13a4fa2a732a6383dcd8c84 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:18.728694 1549 raft_consensus.cc:491] T aad96f2507974023aaa988e6950feb7a P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:18.729136 1549 raft_consensus.cc:513] T aad96f2507974023aaa988e6950feb7a P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:18.730908 1549 leader_election.cc:290] T aad96f2507974023aaa988e6950feb7a P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:18.732697 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet a3886c5f976445cebc95ef1839a74aa9 (table default.Table_35a78676616e4b20906b262c9d0c0d9e [id=e984149e9da444f1b2802e3f35a60fd6]) successfully deleted
I20250114 02:54:18.707904 1205 consensus_queue.cc:260] T fcb58a05f980460884ed81a30a5d59c2 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
W20250114 02:54:18.733330 456 catalog_manager.cc:4852] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet 53a8bb487ad64157ad3baecf5d866eca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 53a8bb487ad64157ad3baecf5d866eca already in progress: creating tablet
W20250114 02:54:18.733866 456 catalog_manager.cc:4852] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet bc46815b4803418c95ea9c039a4fae88 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc46815b4803418c95ea9c039a4fae88 already in progress: creating tablet
I20250114 02:54:18.734643 1205 ts_tablet_manager.cc:1428] T fcb58a05f980460884ed81a30a5d59c2 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.030s user 0.000s sys 0.003s
I20250114 02:54:18.740160 1101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aad96f2507974023aaa988e6950feb7a" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:18.741150 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aad96f2507974023aaa988e6950feb7a" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
W20250114 02:54:18.742283 751 leader_election.cc:343] T aad96f2507974023aaa988e6950feb7a P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:18.750661 1101 raft_consensus.cc:2463] T aad96f2507974023aaa988e6950feb7a P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 0.
I20250114 02:54:18.752568 753 leader_election.cc:304] T aad96f2507974023aaa988e6950feb7a P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters: 525198593c8e47afaee54e74267b87e3
I20250114 02:54:18.753621 1549 raft_consensus.cc:2798] T aad96f2507974023aaa988e6950feb7a P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:18.753980 1549 raft_consensus.cc:491] T aad96f2507974023aaa988e6950feb7a P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:18.754354 1549 raft_consensus.cc:3054] T aad96f2507974023aaa988e6950feb7a P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:18.754895 792 tablet_service.cc:1467] Processing CreateTablet for tablet 7774c2087422492696a58e0075921ae7 (DEFAULT_TABLE table=default.Table_23120ced31c744a0ba995f46d50223a3 [id=a0a99c63c698448499d5a85cb85740bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:18.755957 792 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7774c2087422492696a58e0075921ae7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:18.763127 1206 ts_tablet_manager.cc:1428] T 321b9ad6d1ac4d1c80c7a01faf2015af P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.071s user 0.003s sys 0.001s
I20250114 02:54:18.767599 470 catalog_manager.cc:5526] T de4ca83a69704a368a25954cdc1b9392 P 525198593c8e47afaee54e74267b87e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 525198593c8e47afaee54e74267b87e3 (127.0.107.66). New cstate: current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:18.769919 470 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet de4ca83a69704a368a25954cdc1b9392 (table default.Table_3139140b7eec4849ae715c295fbeaa6b [id=8536eb7f6bf542daa88a8db0e4873456]). Expected version 2 got 0
I20250114 02:54:18.772965 470 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 4bb9b429af92446bbe13894d5dfe4c47 (table default.Table_acb5bf9354be440bbf6034cf1c535dab [id=22b7edcf4b7e4c31b3c23dc44ccf028e]). Expected version 1 got 0
I20250114 02:54:18.776687 1207 tablet_bootstrap.cc:654] T f464eabfbc0945fa8a5583f00c1dab40 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:18.778123 470 catalog_manager.cc:5526] T 9bfd8fd6d2104fb4a6484cb8c98981bd P 525198593c8e47afaee54e74267b87e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 525198593c8e47afaee54e74267b87e3 (127.0.107.66). New cstate: current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:18.795270 470 catalog_manager.cc:5526] T a3fdab330d574c0283ac58bc9e450b87 P 525198593c8e47afaee54e74267b87e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 525198593c8e47afaee54e74267b87e3 (127.0.107.66). New cstate: current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } }
I20250114 02:54:18.797868 1507 raft_consensus.cc:491] T fcb58a05f980460884ed81a30a5d59c2 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:18.798156 1082 tablet_service.cc:1467] Processing CreateTablet for tablet 7774c2087422492696a58e0075921ae7 (DEFAULT_TABLE table=default.Table_23120ced31c744a0ba995f46d50223a3 [id=a0a99c63c698448499d5a85cb85740bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:18.798388 1507 raft_consensus.cc:513] T fcb58a05f980460884ed81a30a5d59c2 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:18.799301 1082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7774c2087422492696a58e0075921ae7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:18.801609 1507 leader_election.cc:290] T fcb58a05f980460884ed81a30a5d59c2 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:18.802860 816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb58a05f980460884ed81a30a5d59c2" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:18.803503 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb58a05f980460884ed81a30a5d59c2" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:18.803565 816 raft_consensus.cc:2463] T fcb58a05f980460884ed81a30a5d59c2 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 0.
I20250114 02:54:18.804035 962 raft_consensus.cc:2463] T fcb58a05f980460884ed81a30a5d59c2 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 0.
I20250114 02:54:18.804873 1042 leader_election.cc:304] T fcb58a05f980460884ed81a30a5d59c2 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:18.807818 1507 raft_consensus.cc:2798] T fcb58a05f980460884ed81a30a5d59c2 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:18.808246 1507 raft_consensus.cc:491] T fcb58a05f980460884ed81a30a5d59c2 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:18.809000 1507 raft_consensus.cc:3054] T fcb58a05f980460884ed81a30a5d59c2 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:18.812875 940 tablet_service.cc:1467] Processing CreateTablet for tablet 7774c2087422492696a58e0075921ae7 (DEFAULT_TABLE table=default.Table_23120ced31c744a0ba995f46d50223a3 [id=a0a99c63c698448499d5a85cb85740bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:18.813818 940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7774c2087422492696a58e0075921ae7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:18.815214 1205 tablet_bootstrap.cc:492] T ba215eb8422d4f879dd0bf5996543962 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:18.819269 1081 tablet_service.cc:1467] Processing CreateTablet for tablet edc69b66bea54625a4b15867b75986b7 (DEFAULT_TABLE table=default.Table_0a0bda55a4a9476c98d961822b8a19bd [id=fb2be5c1ca9444a197f3315ea9c59b09]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:18.820039 1206 tablet_bootstrap.cc:492] T 26328f8e32bc49ed9b224f0e08b9f949 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:18.820320 1081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edc69b66bea54625a4b15867b75986b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:18.820398 791 tablet_service.cc:1467] Processing CreateTablet for tablet edc69b66bea54625a4b15867b75986b7 (DEFAULT_TABLE table=default.Table_0a0bda55a4a9476c98d961822b8a19bd [id=fb2be5c1ca9444a197f3315ea9c59b09]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:18.821408 791 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edc69b66bea54625a4b15867b75986b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:18.823017 942 tablet_service.cc:1467] Processing CreateTablet for tablet edc69b66bea54625a4b15867b75986b7 (DEFAULT_TABLE table=default.Table_0a0bda55a4a9476c98d961822b8a19bd [id=fb2be5c1ca9444a197f3315ea9c59b09]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:18.823455 1549 raft_consensus.cc:513] T aad96f2507974023aaa988e6950feb7a P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:18.824537 942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edc69b66bea54625a4b15867b75986b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:18.828725 1594 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 114) took 1686 ms. Trace:
I20250114 02:54:18.829052 1594 rpcz_store.cc:276] 0114 02:54:17.141831 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:17.147645 (+ 5814us) service_pool.cc:225] Handling call
0114 02:54:18.655037 (+1507392us) ts_tablet_manager.cc:1145] Deleting tablet 961b6a25a13a4fa2a732a6383dcd8c84
0114 02:54:18.655072 (+ 35us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:18.668499 (+ 13427us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:18.723266 (+ 54767us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:18.828655 (+105389us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":101764,"tablet-delete.queue_time_us":1506814}
I20250114 02:54:18.830279 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet 961b6a25a13a4fa2a732a6383dcd8c84 (table default.Table_0943b1ca347042c9a6043006bc566701 [id=e3418787ef0a4e22979fb41d4f94ebb0]) successfully deleted
I20250114 02:54:18.831087 1101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aad96f2507974023aaa988e6950feb7a" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:18.831645 1101 raft_consensus.cc:3054] T aad96f2507974023aaa988e6950feb7a P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:18.837065 1549 leader_election.cc:290] T aad96f2507974023aaa988e6950feb7a P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Requested vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:18.839063 1206 tablet_bootstrap.cc:654] T 26328f8e32bc49ed9b224f0e08b9f949 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:18.798624 470 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet a3fdab330d574c0283ac58bc9e450b87 (table default.Table_ec0eefb64458440ebd57a05a7ad0bb9a [id=b6874c22fe0d4151a7c8ea061f685938]). Expected version 1 got 0
I20250114 02:54:18.840939 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aad96f2507974023aaa988e6950feb7a" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
I20250114 02:54:18.840739 465 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_df476a4b13be4c11a896663f6211624d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:18.842208 465 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_df476a4b13be4c11a896663f6211624d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250114 02:54:18.842271 751 leader_election.cc:343] T aad96f2507974023aaa988e6950feb7a P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:18.846474 1507 raft_consensus.cc:513] T fcb58a05f980460884ed81a30a5d59c2 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
W20250114 02:54:18.851770 456 catalog_manager.cc:4945] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): alter failed for tablet b63dd90f60a04465b948171e74888e79 (table default.Table_1097f09fb10e441fb639f8d7aa12d487 [id=d8c9691818d042d7b49475fa009c4ca1]),no further retry: Not found: Tablet not found: b63dd90f60a04465b948171e74888e79
I20250114 02:54:18.855933 1206 tablet_bootstrap.cc:492] T 26328f8e32bc49ed9b224f0e08b9f949 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:18.856361 1206 ts_tablet_manager.cc:1397] T 26328f8e32bc49ed9b224f0e08b9f949 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.036s user 0.008s sys 0.008s
I20250114 02:54:18.856559 1597 raft_consensus.cc:491] T 9812754c610548d1b43084bbc94fc3e4 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:18.856997 1597 raft_consensus.cc:513] T 9812754c610548d1b43084bbc94fc3e4 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:18.858824 1597 leader_election.cc:290] T 9812754c610548d1b43084bbc94fc3e4 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:18.858603 1206 raft_consensus.cc:357] T 26328f8e32bc49ed9b224f0e08b9f949 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:18.859194 1206 raft_consensus.cc:383] T 26328f8e32bc49ed9b224f0e08b9f949 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:18.859450 1206 raft_consensus.cc:738] T 26328f8e32bc49ed9b224f0e08b9f949 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:18.859979 1205 tablet_bootstrap.cc:654] T ba215eb8422d4f879dd0bf5996543962 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:18.841228 470 catalog_manager.cc:5526] T 15704785f56e43a3af69f181c75877ce P 525198593c8e47afaee54e74267b87e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 525198593c8e47afaee54e74267b87e3 (127.0.107.66). New cstate: current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:18.862546 465 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_d50b0a8b26f644898efcb62a0d72fd1b" } new_table_name: "default.Table_ee814dd266e64286864db0eac9e11664" modify_external_catalogs: true
I20250114 02:54:18.865643 816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9812754c610548d1b43084bbc94fc3e4" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:18.868570 1594 tablet_replica.cc:331] T 99b8acb1fe594bf3825c7b9121018db6 P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:18.869256 1594 raft_consensus.cc:2238] T 99b8acb1fe594bf3825c7b9121018db6 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:18.869657 1594 raft_consensus.cc:2267] T 99b8acb1fe594bf3825c7b9121018db6 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:18.871176 1594 ts_tablet_manager.cc:1905] T 99b8acb1fe594bf3825c7b9121018db6 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:18.857785 1207 tablet_bootstrap.cc:492] T f464eabfbc0945fa8a5583f00c1dab40 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:18.867799 1099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9812754c610548d1b43084bbc94fc3e4" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:18.875862 1099 raft_consensus.cc:2388] T 9812754c610548d1b43084bbc94fc3e4 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 525198593c8e47afaee54e74267b87e3 in current term 1: Already voted for candidate 819f9e3412e54d44abb11d8b3ca877c2 in this term.
I20250114 02:54:18.875895 1595 ts_tablet_manager.cc:1918] T c20aca761313498193e1d79c10705f8f P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:18.876276 1595 log.cc:1198] T c20aca761313498193e1d79c10705f8f P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/c20aca761313498193e1d79c10705f8f
I20250114 02:54:18.876958 1595 ts_tablet_manager.cc:1939] T c20aca761313498193e1d79c10705f8f P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
I20250114 02:54:18.877143 898 leader_election.cc:304] T 9812754c610548d1b43084bbc94fc3e4 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 525198593c8e47afaee54e74267b87e3; no voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:18.877794 1597 raft_consensus.cc:3054] T 9812754c610548d1b43084bbc94fc3e4 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:18.883953 1597 raft_consensus.cc:2743] T 9812754c610548d1b43084bbc94fc3e4 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:18.884521 1206 consensus_queue.cc:260] T 26328f8e32bc49ed9b224f0e08b9f949 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:18.897660 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb58a05f980460884ed81a30a5d59c2" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
I20250114 02:54:18.898162 962 raft_consensus.cc:3054] T fcb58a05f980460884ed81a30a5d59c2 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:18.903707 962 raft_consensus.cc:2463] T fcb58a05f980460884ed81a30a5d59c2 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1.
I20250114 02:54:18.905109 1040 leader_election.cc:304] T fcb58a05f980460884ed81a30a5d59c2 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:18.890277 1207 ts_tablet_manager.cc:1397] T f464eabfbc0945fa8a5583f00c1dab40 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.181s user 0.004s sys 0.007s
I20250114 02:54:18.912393 1595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 117) took 3029 ms. Trace:
I20250114 02:54:18.923713 1206 ts_tablet_manager.cc:1428] T 26328f8e32bc49ed9b224f0e08b9f949 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.067s user 0.006s sys 0.006s
I20250114 02:54:18.936885 471 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 4bb9b429af92446bbe13894d5dfe4c47 (table default.Table_acb5bf9354be440bbf6034cf1c535dab [id=22b7edcf4b7e4c31b3c23dc44ccf028e]). Expected version 1 got 0
I20250114 02:54:18.939218 471 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet a3fdab330d574c0283ac58bc9e450b87 (table default.Table_ec0eefb64458440ebd57a05a7ad0bb9a [id=b6874c22fe0d4151a7c8ea061f685938]). Expected version 1 got 0
I20250114 02:54:18.936712 1595 rpcz_store.cc:276] 0114 02:54:15.883242 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:15.883373 (+ 131us) service_pool.cc:225] Handling call
0114 02:54:18.564449 (+2681076us) ts_tablet_manager.cc:1145] Deleting tablet c20aca761313498193e1d79c10705f8f
0114 02:54:18.564499 (+ 50us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:18.711112 (+146613us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:18.875672 (+164560us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:18.912335 (+ 36663us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2672462}
I20250114 02:54:18.941347 1595 tablet_replica.cc:331] T db24bf3db61941ba9b5e0662db23e748 P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:18.944959 1206 tablet_bootstrap.cc:492] T 55b68586ef8b45439a7dd8dcd2efcd79 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
W20250114 02:54:18.950224 465 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5ac7708550954a209e2a8d217c3b39bd consensus state has no leader
I20250114 02:54:18.951531 465 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_53a42668384c4379998f7d6e5c8b36e7" } new_table_name: "default.Table_60450873174a4c80aa7e2122343d82be" modify_external_catalogs: true
I20250114 02:54:18.956514 1597 raft_consensus.cc:491] T 961b6a25a13a4fa2a732a6383dcd8c84 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:18.956933 1597 raft_consensus.cc:513] T 961b6a25a13a4fa2a732a6383dcd8c84 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:18.961061 816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb58a05f980460884ed81a30a5d59c2" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:18.961617 816 raft_consensus.cc:3054] T fcb58a05f980460884ed81a30a5d59c2 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:18.963095 1595 raft_consensus.cc:2238] T db24bf3db61941ba9b5e0662db23e748 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Raft consensus shutting down.
I20250114 02:54:18.964061 1595 raft_consensus.cc:2267] T db24bf3db61941ba9b5e0662db23e748 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:18.929023 1721 raft_consensus.cc:2798] T fcb58a05f980460884ed81a30a5d59c2 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:18.964936 1721 raft_consensus.cc:695] T fcb58a05f980460884ed81a30a5d59c2 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Becoming Leader. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Running, Role: LEADER
I20250114 02:54:18.965739 1721 consensus_queue.cc:237] T fcb58a05f980460884ed81a30a5d59c2 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:18.933727 1101 raft_consensus.cc:2463] T aad96f2507974023aaa988e6950feb7a P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 1.
I20250114 02:54:18.971180 816 raft_consensus.cc:2463] T fcb58a05f980460884ed81a30a5d59c2 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1.
I20250114 02:54:18.972630 1549 consensus_queue.cc:1035] T 9812754c610548d1b43084bbc94fc3e4 P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:18.944286 1507 leader_election.cc:290] T fcb58a05f980460884ed81a30a5d59c2 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Requested vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:18.980603 469 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet a3fdab330d574c0283ac58bc9e450b87 (table default.Table_ec0eefb64458440ebd57a05a7ad0bb9a [id=b6874c22fe0d4151a7c8ea061f685938]). Expected version 1 got 0
W20250114 02:54:18.953231 1040 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.66:44037, user_credentials={real_user=slave}} blocked reactor thread for 48210.5us
I20250114 02:54:18.981977 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet c20aca761313498193e1d79c10705f8f (table default.Table_6a4fe39298514187b65def5841d3a992 [id=5c5a7974ad2f4e58ba5c47a6eeaaac62]) successfully deleted
I20250114 02:54:18.982264 469 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet de4ca83a69704a368a25954cdc1b9392 (table default.Table_3139140b7eec4849ae715c295fbeaa6b [id=8536eb7f6bf542daa88a8db0e4873456]). Expected version 2 got 0
I20250114 02:54:18.982522 1595 ts_tablet_manager.cc:1905] T db24bf3db61941ba9b5e0662db23e748 P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:18.991770 753 leader_election.cc:304] T aad96f2507974023aaa988e6950feb7a P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters: 525198593c8e47afaee54e74267b87e3
I20250114 02:54:18.992802 1549 raft_consensus.cc:2798] T aad96f2507974023aaa988e6950feb7a P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:18.993237 1549 raft_consensus.cc:695] T aad96f2507974023aaa988e6950feb7a P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 LEADER]: Becoming Leader. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Running, Role: LEADER
I20250114 02:54:18.993974 1549 consensus_queue.cc:237] T aad96f2507974023aaa988e6950feb7a P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
W20250114 02:54:19.003646 465 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 38fe8829d98547fb907ef78099df3d7f has no consensus state
I20250114 02:54:19.004519 464 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet de4ca83a69704a368a25954cdc1b9392 (table default.Table_3139140b7eec4849ae715c295fbeaa6b [id=8536eb7f6bf542daa88a8db0e4873456]). Expected version 2 got 0
I20250114 02:54:18.912798 1207 raft_consensus.cc:357] T f464eabfbc0945fa8a5583f00c1dab40 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:19.004909 1207 raft_consensus.cc:383] T f464eabfbc0945fa8a5583f00c1dab40 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:19.004648 465 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_910b0ec1594a4dfcbb342055cab3eb65"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:19.005196 1207 raft_consensus.cc:738] T f464eabfbc0945fa8a5583f00c1dab40 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
W20250114 02:54:19.005970 465 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_910b0ec1594a4dfcbb342055cab3eb65 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:19.006107 464 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet f464eabfbc0945fa8a5583f00c1dab40 (table default.Table_4fd891b63924429085d78f78cb177f0e [id=d36c71e7562b4ff1bc8be5cf2955000e]). Expected version 1 got 0
I20250114 02:54:19.005730 1207 consensus_queue.cc:260] T f464eabfbc0945fa8a5583f00c1dab40 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:19.007120 464 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet a3fdab330d574c0283ac58bc9e450b87 (table default.Table_ec0eefb64458440ebd57a05a7ad0bb9a [id=b6874c22fe0d4151a7c8ea061f685938]). Expected version 1 got 0
I20250114 02:54:19.007997 1207 ts_tablet_manager.cc:1428] T f464eabfbc0945fa8a5583f00c1dab40 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.097s user 0.004s sys 0.000s
I20250114 02:54:19.010304 1652 consensus_queue.cc:1035] T 9812754c610548d1b43084bbc94fc3e4 P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:19.025848 1206 tablet_bootstrap.cc:654] T 55b68586ef8b45439a7dd8dcd2efcd79 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:19.026346 1594 ts_tablet_manager.cc:1918] T 99b8acb1fe594bf3825c7b9121018db6 P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:19.026098 465 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_dc087fb027f74c4a8bd5dede1142d97d" } modify_external_catalogs: true
I20250114 02:54:19.026692 1594 log.cc:1198] T 99b8acb1fe594bf3825c7b9121018db6 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/99b8acb1fe594bf3825c7b9121018db6
I20250114 02:54:19.027096 465 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_dc087fb027f74c4a8bd5dede1142d97d" } modify_external_catalogs: true
I20250114 02:54:19.027392 1594 ts_tablet_manager.cc:1939] T 99b8acb1fe594bf3825c7b9121018db6 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:19.049919 467 catalog_manager.cc:5526] T 9812754c610548d1b43084bbc94fc3e4 P 819f9e3412e54d44abb11d8b3ca877c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65). New cstate: current_term: 1 leader_uuid: "819f9e3412e54d44abb11d8b3ca877c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:19.052311 467 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet de4ca83a69704a368a25954cdc1b9392 (table default.Table_3139140b7eec4849ae715c295fbeaa6b [id=8536eb7f6bf542daa88a8db0e4873456]). Expected version 2 got 0
I20250114 02:54:19.056723 1597 leader_election.cc:290] T 961b6a25a13a4fa2a732a6383dcd8c84 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
W20250114 02:54:19.059145 898 leader_election.cc:343] T 961b6a25a13a4fa2a732a6383dcd8c84 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Not found: Tablet not found: 961b6a25a13a4fa2a732a6383dcd8c84
I20250114 02:54:19.070813 467 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet f464eabfbc0945fa8a5583f00c1dab40 (table default.Table_4fd891b63924429085d78f78cb177f0e [id=d36c71e7562b4ff1bc8be5cf2955000e]). Expected version 1 got 0
I20250114 02:54:19.080660 467 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet a3fdab330d574c0283ac58bc9e450b87 (table default.Table_ec0eefb64458440ebd57a05a7ad0bb9a [id=b6874c22fe0d4151a7c8ea061f685938]). Expected version 1 got 0
I20250114 02:54:19.081075 465 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet f0bd9bd5a17d418d8082ed0ebd1bf4cc
I20250114 02:54:19.085455 467 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet aad96f2507974023aaa988e6950feb7a (table default.Table_9676926a50eb42349a2d609b30566e9a [id=f38063ebf94d4084ab0174428503dc29]). Expected version 2 got 0
I20250114 02:54:19.086141 791 tablet_service.cc:1514] Processing DeleteTablet for tablet f0bd9bd5a17d418d8082ed0ebd1bf4cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:19 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:19.087476 466 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet 00080bea6c8e46e584ccba6dcb0e221b
I20250114 02:54:19.088840 466 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:33540 (request call id 175) took 1770 ms. Trace:
I20250114 02:54:19.090582 465 master_service.cc:946] ReplaceTablet: received request to replace tablet 46360f726fad4534a2daf67581346fc1 from {username='slave'} at 127.0.0.1:33540
I20250114 02:54:19.091094 1594 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 113) took 1979 ms. Trace:
I20250114 02:54:19.091496 1594 rpcz_store.cc:276] 0114 02:54:17.111317 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:17.215009 (+103692us) service_pool.cc:225] Handling call
0114 02:54:18.868439 (+1653430us) ts_tablet_manager.cc:1145] Deleting tablet 99b8acb1fe594bf3825c7b9121018db6
0114 02:54:18.868478 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:18.956777 (+ 88299us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:19.026141 (+ 69364us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:19.091042 (+ 64901us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1641708}
I20250114 02:54:19.093549 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet 99b8acb1fe594bf3825c7b9121018db6 (table default.Table_661ed49bcbbb476aa68742b72604d95f [id=00383db95e10461a83394c5bd94ca94a]) successfully deleted
I20250114 02:54:19.089210 466 rpcz_store.cc:276] 0114 02:54:17.318368 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:17.318487 (+ 119us) service_pool.cc:225] Handling call
0114 02:54:17.408768 (+ 90281us) spinlock_profiling.cc:243] Waited 89.1 ms on lock 0x7b54000316d4. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616cad50 00007f646174df2a 00007f64633f2dec 00007f64634ec616 00007f6461254b44 00007f6461254ad2 00007f6461254a01 00007f646125497c 00007f6461253c32 00007f645f512dfc 00007f645f512236
0114 02:54:17.416262 (+ 7494us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0114 02:54:17.422061 (+ 5799us) catalog_manager.cc:2753] Modifying in-memory table state
0114 02:54:17.422331 (+ 270us) catalog_manager.cc:2762] Locking tablets
0114 02:54:18.309323 (+886992us) catalog_manager.cc:2776] Removing table and tablets from system table
0114 02:54:18.416602 (+107279us) catalog_manager.cc:2794] Removing table from by-name map
0114 02:54:18.416795 (+ 193us) catalog_manager.cc:2810] Committing in-memory state
0114 02:54:19.087343 (+670548us) catalog_manager.cc:2814] Aborting table tasks
0114 02:54:19.088771 (+ 1428us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:18.371689 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:18.372001 (+ 312us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:18.372022 (+ 21us) write_op.cc:435] Acquired schema lock
0114 02:54:18.372033 (+ 11us) tablet.cc:659] Decoding operations
0114 02:54:18.372352 (+ 319us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:18.372387 (+ 35us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:18.372406 (+ 19us) tablet.cc:682] Acquiring locks for 2 operations
0114 02:54:18.372625 (+ 219us) tablet.cc:698] Row locks acquired
0114 02:54:18.372636 (+ 11us) write_op.cc:260] PREPARE: finished
0114 02:54:18.372723 (+ 87us) write_op.cc:270] Start()
0114 02:54:18.372792 (+ 69us) write_op.cc:276] Timestamp: P: 1736823258372662 usec, L: 0
0114 02:54:18.372805 (+ 13us) op_driver.cc:348] REPLICATION: starting
0114 02:54:18.373200 (+ 395us) log.cc:844] Serialized 728 byte log entry
0114 02:54:18.414116 (+ 40916us) op_driver.cc:464] REPLICATION: finished
0114 02:54:18.414778 (+ 662us) write_op.cc:301] APPLY: starting
0114 02:54:18.414830 (+ 52us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:18.414953 (+ 123us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:18.415037 (+ 84us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:18.415267 (+ 230us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:18.415396 (+ 129us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0114 02:54:18.415438 (+ 42us) write_op.cc:312] APPLY: finished
0114 02:54:18.415590 (+ 152us) log.cc:844] Serialized 29 byte log entry
0114 02:54:18.415720 (+ 130us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:18.415784 (+ 64us) write_op.cc:454] Released schema lock
0114 02:54:18.415813 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:18.427535 (+ 11722us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:18.427558 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":205014912,"child_traces":[["op",{"apply.queue_time_us":536,"apply.run_cpu_time_us":1194,"apply.run_wall_time_us":1194,"num_ops":2,"prepare.queue_time_us":60679,"prepare.run_cpu_time_us":1815,"prepare.run_wall_time_us":55833,"raft.queue_time_us":89,"raft.run_cpu_time_us":647,"raft.run_wall_time_us":647,"replication_time_us":41225}]]}
I20250114 02:54:19.097420 466 master_service.cc:946] ReplaceTablet: received request to replace tablet ba215eb8422d4f879dd0bf5996543962 from {username='slave'} at 127.0.0.1:33540
I20250114 02:54:19.102782 791 tablet_service.cc:1514] Processing DeleteTablet for tablet 00080bea6c8e46e584ccba6dcb0e221b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:17 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:19.104020 1081 tablet_service.cc:1514] Processing DeleteTablet for tablet 00080bea6c8e46e584ccba6dcb0e221b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:17 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:19.104403 1082 tablet_service.cc:1514] Processing DeleteTablet for tablet f0bd9bd5a17d418d8082ed0ebd1bf4cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:19 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:19.107112 1206 tablet_bootstrap.cc:492] T 55b68586ef8b45439a7dd8dcd2efcd79 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:19.107501 1206 ts_tablet_manager.cc:1397] T 55b68586ef8b45439a7dd8dcd2efcd79 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.163s user 0.004s sys 0.004s
I20250114 02:54:19.108026 1594 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 116) took 1842 ms. Trace:
I20250114 02:54:19.110322 462 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: default.Table_3652257c54c3434da1b93eb0481a0df7 [id=5e6aa799d6444bad968f29327ebace9b] alter complete (version 2)
I20250114 02:54:19.111439 1594 rpcz_store.cc:276] 0114 02:54:17.265564 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:17.265669 (+ 105us) service_pool.cc:225] Handling call
0114 02:54:19.107135 (+1841466us) ts_tablet_manager.cc:1145] Deleting tablet 66bfa92db9874750b3c340e31a06c412
0114 02:54:19.107186 (+ 51us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:19.107942 (+ 756us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1840950}
W20250114 02:54:19.112556 458 catalog_manager.cc:4852] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet 66bfa92db9874750b3c340e31a06c412 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 66bfa92db9874750b3c340e31a06c412 already in progress: creating tablet
I20250114 02:54:19.120465 1594 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 117) took 1829 ms. Trace:
I20250114 02:54:19.120795 1594 rpcz_store.cc:276] 0114 02:54:17.290573 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:17.290700 (+ 127us) service_pool.cc:225] Handling call
0114 02:54:19.120141 (+1829441us) ts_tablet_manager.cc:1145] Deleting tablet b4740f6b3a6741368ba68907ad065279
0114 02:54:19.120182 (+ 41us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:19.120395 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1828798}
W20250114 02:54:19.121867 458 catalog_manager.cc:4839] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet b4740f6b3a6741368ba68907ad065279 because the tablet was not found. No further retry: Not found: Tablet not found: b4740f6b3a6741368ba68907ad065279
I20250114 02:54:19.122165 1205 tablet_bootstrap.cc:492] T ba215eb8422d4f879dd0bf5996543962 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:19.122517 1205 ts_tablet_manager.cc:1397] T ba215eb8422d4f879dd0bf5996543962 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.342s user 0.009s sys 0.009s
I20250114 02:54:19.124572 1205 raft_consensus.cc:357] T ba215eb8422d4f879dd0bf5996543962 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:19.125173 1205 raft_consensus.cc:383] T ba215eb8422d4f879dd0bf5996543962 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:19.125391 1205 raft_consensus.cc:738] T ba215eb8422d4f879dd0bf5996543962 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:19.125865 1205 consensus_queue.cc:260] T ba215eb8422d4f879dd0bf5996543962 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:19.127676 1205 ts_tablet_manager.cc:1428] T ba215eb8422d4f879dd0bf5996543962 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20250114 02:54:19.128778 1205 tablet_bootstrap.cc:492] T aad96f2507974023aaa988e6950feb7a P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:19.128705 1594 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 119) took 1704 ms. Trace:
I20250114 02:54:19.129014 1594 rpcz_store.cc:276] 0114 02:54:17.424048 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:17.424314 (+ 266us) service_pool.cc:225] Handling call
0114 02:54:19.128463 (+1704149us) ts_tablet_manager.cc:1145] Deleting tablet 83b5b36a26e74668998a19fa0ddf7188
0114 02:54:19.128503 (+ 40us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:19.128652 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1688523}
W20250114 02:54:19.130079 458 catalog_manager.cc:4839] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet 83b5b36a26e74668998a19fa0ddf7188 because the tablet was not found. No further retry: Not found: Tablet not found: 83b5b36a26e74668998a19fa0ddf7188
I20250114 02:54:19.133723 468 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: default.Table_d90b2995b13b4a8fb7e50cc4b79cbf84 [id=f8e40e0aa44b4157ac3438d11685b2e7] alter complete (version 1)
I20250114 02:54:19.109776 1206 raft_consensus.cc:357] T 55b68586ef8b45439a7dd8dcd2efcd79 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:19.138180 1206 raft_consensus.cc:383] T 55b68586ef8b45439a7dd8dcd2efcd79 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:19.138342 1594 tablet_replica.cc:331] T ed2d137579b8467da5eed4ea2f619a9a P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:19.138442 1206 raft_consensus.cc:738] T 55b68586ef8b45439a7dd8dcd2efcd79 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:19.139139 1206 consensus_queue.cc:260] T 55b68586ef8b45439a7dd8dcd2efcd79 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:19.141527 1594 raft_consensus.cc:2238] T ed2d137579b8467da5eed4ea2f619a9a P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:19.141983 1594 raft_consensus.cc:2267] T ed2d137579b8467da5eed4ea2f619a9a P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:19.143599 791 tablet_service.cc:1514] Processing DeleteTablet for tablet b4740f6b3a6741368ba68907ad065279 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:14 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:19.145279 798 tablet_service.cc:1514] Processing DeleteTablet for tablet 83b5b36a26e74668998a19fa0ddf7188 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:12 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:19.145530 795 tablet_service.cc:1514] Processing DeleteTablet for tablet 290ad07ab8be4791a39194909e000843 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:13 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:19.145556 792 tablet_service.cc:1514] Processing DeleteTablet for tablet 9589afd0094545c1801b7a0b5eb8698f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:13 UTC) from {username='slave'} at 127.0.0.1:53078
W20250114 02:54:19.146654 462 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.65:49805 (request call id 42) took 2600 ms (client timeout 1000 ms). Trace:
I20250114 02:54:19.147833 791 tablet_service.cc:1514] Processing DeleteTablet for tablet 276942e5e54246c79f0d4f08297f58ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:14 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:19.148370 792 tablet_service.cc:1514] Processing DeleteTablet for tablet b63dd90f60a04465b948171e74888e79 with delete_type TABLET_DATA_DELETED (replaced by tablet 59a46bc865cc48df8e98598764fd1b54) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:19.148087 795 tablet_service.cc:1514] Processing DeleteTablet for tablet 96b06fe4173343dabd726d253b9f9b5f with delete_type TABLET_DATA_DELETED (replaced by tablet e15e6e52d7294aeeb4463b3a76022ec3) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:19.150892 471 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: default.Table_58e014a2cfee480184ff3c66b1768cf6 [id=5745501ecd7b4cecaca3d35502a6cc40] alter complete (version 2)
I20250114 02:54:19.161965 1595 ts_tablet_manager.cc:1918] T db24bf3db61941ba9b5e0662db23e748 P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:19.162362 1595 log.cc:1198] T db24bf3db61941ba9b5e0662db23e748 P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/db24bf3db61941ba9b5e0662db23e748
I20250114 02:54:19.163129 1595 ts_tablet_manager.cc:1939] T db24bf3db61941ba9b5e0662db23e748 P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
I20250114 02:54:19.164181 1594 ts_tablet_manager.cc:1905] T ed2d137579b8467da5eed4ea2f619a9a P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:19.165938 1101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "961b6a25a13a4fa2a732a6383dcd8c84" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:19.166658 1101 raft_consensus.cc:2388] T 961b6a25a13a4fa2a732a6383dcd8c84 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 525198593c8e47afaee54e74267b87e3 in current term 1: Already voted for candidate 819f9e3412e54d44abb11d8b3ca877c2 in this term.
I20250114 02:54:19.171114 1206 ts_tablet_manager.cc:1428] T 55b68586ef8b45439a7dd8dcd2efcd79 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.063s user 0.007s sys 0.000s
I20250114 02:54:19.172124 466 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet ba215eb8422d4f879dd0bf5996543962
I20250114 02:54:19.175413 466 catalog_manager.cc:6596] ReplaceTablet: tablet ba215eb8422d4f879dd0bf5996543962 deleted and replaced by tablet f40c66314342453b898da3ac0e7e86bf
I20250114 02:54:19.175369 795 tablet_service.cc:1514] Processing DeleteTablet for tablet ba215eb8422d4f879dd0bf5996543962 with delete_type TABLET_DATA_DELETED (replaced by tablet f40c66314342453b898da3ac0e7e86bf) from {username='slave'} at 127.0.0.1:53078
W20250114 02:54:19.176338 470 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.66:50663 (request call id 41) took 1627 ms (client timeout 1000 ms). Trace:
I20250114 02:54:19.171283 469 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: default.Table_acb5bf9354be440bbf6034cf1c535dab [id=22b7edcf4b7e4c31b3c23dc44ccf028e] alter complete (version 1)
W20250114 02:54:19.176646 470 rpcz_store.cc:269] 0114 02:54:17.548956 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:18.007059 (+458103us) service_pool.cc:225] Handling call
0114 02:54:18.014130 (+ 7071us) spinlock_profiling.cc:243] Waited 3.7 ms on lock 0x7b54000316d4. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616cad50 00007f646174df2a 00007f64633f2dec 00007f64634eadee 00007f6461248fc4 00007f6461248f52 00007f6461248e81 00007f6461248dfc 00007f64612480b2 00007f645f512dfc 00007f645f512236
0114 02:54:19.176260 (+1162130us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:18.864366 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:18.864647 (+ 281us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:18.864667 (+ 20us) write_op.cc:435] Acquired schema lock
0114 02:54:18.864679 (+ 12us) tablet.cc:659] Decoding operations
0114 02:54:18.865061 (+ 382us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:18.865095 (+ 34us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:18.865112 (+ 17us) tablet.cc:682] Acquiring locks for 4 operations
0114 02:54:18.865351 (+ 239us) tablet.cc:698] Row locks acquired
0114 02:54:18.865363 (+ 12us) write_op.cc:260] PREPARE: finished
0114 02:54:18.865458 (+ 95us) write_op.cc:270] Start()
0114 02:54:18.865522 (+ 64us) write_op.cc:276] Timestamp: P: 1736823258865408 usec, L: 0
0114 02:54:18.865535 (+ 13us) op_driver.cc:348] REPLICATION: starting
0114 02:54:18.865923 (+ 388us) log.cc:844] Serialized 1662 byte log entry
0114 02:54:18.915189 (+ 49266us) op_driver.cc:464] REPLICATION: finished
0114 02:54:18.915758 (+ 569us) write_op.cc:301] APPLY: starting
0114 02:54:18.915814 (+ 56us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:18.915944 (+ 130us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:18.916048 (+ 104us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:18.916427 (+ 379us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:18.916562 (+ 135us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0114 02:54:18.916603 (+ 41us) write_op.cc:312] APPLY: finished
0114 02:54:18.916808 (+ 205us) log.cc:844] Serialized 41 byte log entry
0114 02:54:18.916943 (+ 135us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:18.917015 (+ 72us) write_op.cc:454] Released schema lock
0114 02:54:18.917061 (+ 46us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:18.917803 (+ 742us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:18.917827 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":118738,"reported_tablets":15,"spinlock_wait_cycles":8513792,"child_traces":[["op",{"apply.queue_time_us":442,"apply.run_cpu_time_us":1465,"apply.run_wall_time_us":1464,"num_ops":4,"prepare.queue_time_us":1136,"prepare.run_cpu_time_us":1810,"prepare.run_wall_time_us":2987,"raft.queue_time_us":74,"raft.run_cpu_time_us":18,"raft.run_wall_time_us":18,"replication_time_us":49559}]]}
I20250114 02:54:19.183966 792 tablet_service.cc:1514] Processing DeleteTablet for tablet db24bf3db61941ba9b5e0662db23e748 with delete_type TABLET_DATA_DELETED (replaced by tablet ddfbab181b1f474192c664cd0316037f) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:19.186147 795 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ecf782bb3d54528aa996abca1ea3a15 with delete_type TABLET_DATA_DELETED (replaced by tablet 66bfa92db9874750b3c340e31a06c412) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:19.189114 1206 tablet_bootstrap.cc:492] T 266da813b4514329be0ad97d07b7114f P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:19.192440 938 tablet_service.cc:1514] Processing DeleteTablet for tablet 00080bea6c8e46e584ccba6dcb0e221b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:17 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:19.193305 1702 tablet_replica.cc:331] T 00080bea6c8e46e584ccba6dcb0e221b P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:19.194129 1702 raft_consensus.cc:2238] T 00080bea6c8e46e584ccba6dcb0e221b P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:19.194568 1702 raft_consensus.cc:2267] T 00080bea6c8e46e584ccba6dcb0e221b P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:19.196516 1702 ts_tablet_manager.cc:1905] T 00080bea6c8e46e584ccba6dcb0e221b P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:19.198346 795 tablet_service.cc:1514] Processing DeleteTablet for tablet 961b6a25a13a4fa2a732a6383dcd8c84 with delete_type TABLET_DATA_DELETED (replaced by tablet 99b940d872ce4781907517c40fe08781) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:19.199492 1082 tablet_service.cc:1514] Processing DeleteTablet for tablet ba215eb8422d4f879dd0bf5996543962 with delete_type TABLET_DATA_DELETED (replaced by tablet f40c66314342453b898da3ac0e7e86bf) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:19.199709 1081 tablet_service.cc:1514] Processing DeleteTablet for tablet 268a12df225e4ba59d4a84c2869d4661 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:10 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:19.200660 1081 tablet_service.cc:1514] Processing DeleteTablet for tablet ed2d137579b8467da5eed4ea2f619a9a with delete_type TABLET_DATA_DELETED (replaced by tablet 0304df55431e42e1a999ed17dc35e6c0) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:19.201589 1081 tablet_service.cc:1514] Processing DeleteTablet for tablet 9589afd0094545c1801b7a0b5eb8698f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:13 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:19.203509 1081 tablet_service.cc:1514] Processing DeleteTablet for tablet c20aca761313498193e1d79c10705f8f with delete_type TABLET_DATA_DELETED (replaced by tablet bc46815b4803418c95ea9c039a4fae88) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:19.205253 1081 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ecf782bb3d54528aa996abca1ea3a15 with delete_type TABLET_DATA_DELETED (replaced by tablet 66bfa92db9874750b3c340e31a06c412) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:19.176194 466 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_80da487b7f04491888501948c55ed09f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:19.211509 470 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_1ec80cfdcee64e98beccc30fec0afff8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:19.195428 471 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.65:49805 (request call id 43) took 1657 ms (client timeout 1000 ms). Trace:
W20250114 02:54:19.213135 463 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.67:34825 (request call id 39) took 2160 ms (client timeout 1000 ms). Trace:
W20250114 02:54:19.213052 471 rpcz_store.cc:269] 0114 02:54:17.538244 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:17.587079 (+ 48835us) service_pool.cc:225] Handling call
0114 02:54:18.018165 (+431086us) spinlock_profiling.cc:243] Waited 290 ms on lock 0x7b54000316d4. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616cad50 00007f646174df2a 00007f64633f2dec 00007f64634eadee 00007f6461248fc4 00007f6461248f52 00007f6461248e81 00007f6461248dfc 00007f64612480b2 00007f645f512dfc 00007f645f512236
0114 02:54:19.195367 (+1177202us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:19.117110 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:19.117440 (+ 330us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:19.117461 (+ 21us) write_op.cc:435] Acquired schema lock
0114 02:54:19.117473 (+ 12us) tablet.cc:659] Decoding operations
0114 02:54:19.117688 (+ 215us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:19.117721 (+ 33us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:19.117740 (+ 19us) tablet.cc:682] Acquiring locks for 1 operations
0114 02:54:19.117857 (+ 117us) tablet.cc:698] Row locks acquired
0114 02:54:19.117868 (+ 11us) write_op.cc:260] PREPARE: finished
0114 02:54:19.117956 (+ 88us) write_op.cc:270] Start()
0114 02:54:19.118023 (+ 67us) write_op.cc:276] Timestamp: P: 1736823259117881 usec, L: 0
0114 02:54:19.118037 (+ 14us) op_driver.cc:348] REPLICATION: starting
0114 02:54:19.118405 (+ 368us) log.cc:844] Serialized 356 byte log entry
0114 02:54:19.139458 (+ 21053us) op_driver.cc:464] REPLICATION: finished
0114 02:54:19.149486 (+ 10028us) write_op.cc:301] APPLY: starting
0114 02:54:19.149557 (+ 71us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:19.149638 (+ 81us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:19.149717 (+ 79us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:19.149837 (+ 120us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:19.149947 (+ 110us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:19.149982 (+ 35us) write_op.cc:312] APPLY: finished
0114 02:54:19.150125 (+ 143us) log.cc:844] Serialized 23 byte log entry
0114 02:54:19.150243 (+ 118us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:19.150292 (+ 49us) write_op.cc:454] Released schema lock
0114 02:54:19.150325 (+ 33us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:19.150523 (+ 198us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:19.150551 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":5569,"reported_tablets":21,"spinlock_wait_cycles":667345152,"child_traces":[["op",{"apply.queue_time_us":9895,"apply.run_cpu_time_us":1018,"apply.run_wall_time_us":1018,"num_ops":1,"prepare.queue_time_us":97,"prepare.run_cpu_time_us":1467,"prepare.run_wall_time_us":1469,"replication_time_us":21351}]]}
I20250114 02:54:19.226270 1205 tablet_bootstrap.cc:654] T aad96f2507974023aaa988e6950feb7a P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
W20250114 02:54:19.228684 466 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_80da487b7f04491888501948c55ed09f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:19.231140 1206 tablet_bootstrap.cc:654] T 266da813b4514329be0ad97d07b7114f P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
W20250114 02:54:19.231987 468 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.66:50663 (request call id 40) took 2691 ms (client timeout 1000 ms). Trace:
I20250114 02:54:19.235711 1207 tablet_bootstrap.cc:492] T ba215eb8422d4f879dd0bf5996543962 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
W20250114 02:54:19.236047 469 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.67:34825 (request call id 40) took 1049 ms (client timeout 1000 ms). Trace:
I20250114 02:54:19.240942 1595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 118) took 3316 ms. Trace:
I20250114 02:54:19.244530 1595 rpcz_store.cc:276] 0114 02:54:15.924474 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:15.924606 (+ 132us) service_pool.cc:225] Handling call
0114 02:54:18.941185 (+3016579us) ts_tablet_manager.cc:1145] Deleting tablet db24bf3db61941ba9b5e0662db23e748
0114 02:54:18.941233 (+ 48us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:19.106667 (+165434us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:19.161693 (+ 55026us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:19.240885 (+ 79192us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":65292,"tablet-delete.queue_time_us":3015892}
I20250114 02:54:19.247217 1732 tablet.cc:1721] T de4ca83a69704a368a25954cdc1b9392 P 525198593c8e47afaee54e74267b87e3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250114 02:54:19.213909 463 rpcz_store.cc:269] 0114 02:54:17.052973 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:17.121368 (+ 68395us) service_pool.cc:225] Handling call
0114 02:54:19.212953 (+2091585us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:18.542808 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:18.543115 (+ 307us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:18.543136 (+ 21us) write_op.cc:435] Acquired schema lock
0114 02:54:18.543148 (+ 12us) tablet.cc:659] Decoding operations
0114 02:54:18.543370 (+ 222us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:18.543400 (+ 30us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:18.543420 (+ 20us) tablet.cc:682] Acquiring locks for 1 operations
0114 02:54:18.543549 (+ 129us) tablet.cc:698] Row locks acquired
0114 02:54:18.543562 (+ 13us) write_op.cc:260] PREPARE: finished
0114 02:54:18.543652 (+ 90us) write_op.cc:270] Start()
0114 02:54:18.543718 (+ 66us) write_op.cc:276] Timestamp: P: 1736823258543571 usec, L: 0
0114 02:54:18.543732 (+ 14us) op_driver.cc:348] REPLICATION: starting
0114 02:54:18.544151 (+ 419us) log.cc:844] Serialized 530 byte log entry
0114 02:54:18.756429 (+212278us) op_driver.cc:464] REPLICATION: finished
0114 02:54:18.758601 (+ 2172us) write_op.cc:301] APPLY: starting
0114 02:54:18.758645 (+ 44us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:18.758731 (+ 86us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:18.758815 (+ 84us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:18.758968 (+ 153us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:18.759080 (+ 112us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:18.759128 (+ 48us) write_op.cc:312] APPLY: finished
0114 02:54:18.759278 (+ 150us) log.cc:844] Serialized 23 byte log entry
0114 02:54:18.759407 (+ 129us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:18.759454 (+ 47us) write_op.cc:454] Released schema lock
0114 02:54:18.759488 (+ 34us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":22,"child_traces":[["op",{"apply.queue_time_us":2059,"apply.run_cpu_time_us":1056,"apply.run_wall_time_us":1056,"num_ops":1,"prepare.queue_time_us":131911,"prepare.run_cpu_time_us":1526,"prepare.run_wall_time_us":3491,"replication_time_us":212622,"thread_start_us":49029,"threads_started":1}]]}
I20250114 02:54:19.247546 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet db24bf3db61941ba9b5e0662db23e748 (table default.Table_494541d472dc48d0bef33a634c1180fa [id=5ae72503278349c0bbe315174c896074]) successfully deleted
I20250114 02:54:19.249822 1595 tablet_replica.cc:331] T 96b06fe4173343dabd726d253b9f9b5f P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
W20250114 02:54:19.232371 468 rpcz_store.cc:269] 0114 02:54:16.540198 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:16.552113 (+ 11915us) service_pool.cc:225] Handling call
0114 02:54:16.620841 (+ 68728us) spinlock_profiling.cc:243] Waited 64.9 ms on lock 0x7b54000316d4. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616cad50 00007f646174df2a 00007f64633f2dec 00007f64634eadee 00007f6461248fc4 00007f6461248f52 00007f6461248e81 00007f6461248dfc 00007f64612480b2 00007f645f512dfc 00007f645f512236
0114 02:54:19.231915 (+2611074us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:17.581619 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:17.581943 (+ 324us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:17.581964 (+ 21us) write_op.cc:435] Acquired schema lock
0114 02:54:17.581976 (+ 12us) tablet.cc:659] Decoding operations
0114 02:54:17.582199 (+ 223us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:17.582245 (+ 46us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:17.582266 (+ 21us) tablet.cc:682] Acquiring locks for 1 operations
0114 02:54:17.582390 (+ 124us) tablet.cc:698] Row locks acquired
0114 02:54:17.582401 (+ 11us) write_op.cc:260] PREPARE: finished
0114 02:54:17.582498 (+ 97us) write_op.cc:270] Start()
0114 02:54:17.582576 (+ 78us) write_op.cc:276] Timestamp: P: 1736823257582414 usec, L: 0
0114 02:54:17.582590 (+ 14us) op_driver.cc:348] REPLICATION: starting
0114 02:54:17.583009 (+ 419us) log.cc:844] Serialized 356 byte log entry
0114 02:54:17.890893 (+307884us) op_driver.cc:464] REPLICATION: finished
0114 02:54:17.892181 (+ 1288us) write_op.cc:301] APPLY: starting
0114 02:54:17.892234 (+ 53us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:17.892354 (+ 120us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:17.892450 (+ 96us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:17.892607 (+ 157us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:17.892718 (+ 111us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:17.892759 (+ 41us) write_op.cc:312] APPLY: finished
0114 02:54:17.892926 (+ 167us) log.cc:844] Serialized 23 byte log entry
0114 02:54:17.893090 (+ 164us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:17.893154 (+ 64us) write_op.cc:454] Released schema lock
0114 02:54:17.893181 (+ 27us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:17.893369 (+ 188us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:17.893390 (+ 21us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":6166,"reported_tablets":19,"spinlock_wait_cycles":149164288,"child_traces":[["op",{"apply.queue_time_us":1150,"apply.run_cpu_time_us":1147,"apply.run_wall_time_us":1155,"num_ops":1,"prepare.queue_time_us":229497,"prepare.run_cpu_time_us":1717,"prepare.run_wall_time_us":40234,"raft.queue_time_us":91,"raft.run_cpu_time_us":625,"raft.run_wall_time_us":626,"replication_time_us":308206}]]}
I20250114 02:54:19.252120 1595 raft_consensus.cc:2238] T 96b06fe4173343dabd726d253b9f9b5f P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:19.252642 1595 raft_consensus.cc:2267] T 96b06fe4173343dabd726d253b9f9b5f P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:19.253065 464 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_11c84abe27d74b67bfbd28c249c78b03" } new_table_name: "default.Table_428bd60f16824ac2b8bfaad6819f76e1" modify_external_catalogs: true
I20250114 02:54:19.256796 1733 tablet.cc:1721] T de4ca83a69704a368a25954cdc1b9392 P 819f9e3412e54d44abb11d8b3ca877c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250114 02:54:19.259096 1595 ts_tablet_manager.cc:1905] T 96b06fe4173343dabd726d253b9f9b5f P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:19.257994 468 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_9a7ce95817ea4e4fb4bc3d8ccb6f5e8c" } new_table_name: "default.Table_0518d8bed7304c86aff32e05e6f07c0d" modify_external_catalogs: true
W20250114 02:54:19.236438 469 rpcz_store.cc:269] 0114 02:54:18.186780 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:18.224765 (+ 37985us) service_pool.cc:225] Handling call
0114 02:54:18.234600 (+ 9835us) spinlock_profiling.cc:243] Waited 4.58 ms on lock 0x7b54000316d4. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616cad50 00007f646174df2a 00007f64633f2dec 00007f64634eadee 00007f6461248fc4 00007f6461248f52 00007f6461248e81 00007f6461248dfc 00007f64612480b2 00007f645f512dfc 00007f645f512236
0114 02:54:19.235983 (+1001383us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:19.105543 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:19.105922 (+ 379us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:19.105947 (+ 25us) write_op.cc:435] Acquired schema lock
0114 02:54:19.105963 (+ 16us) tablet.cc:659] Decoding operations
0114 02:54:19.106209 (+ 246us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:19.106252 (+ 43us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:19.106273 (+ 21us) tablet.cc:682] Acquiring locks for 1 operations
0114 02:54:19.106412 (+ 139us) tablet.cc:698] Row locks acquired
0114 02:54:19.106424 (+ 12us) write_op.cc:260] PREPARE: finished
0114 02:54:19.106523 (+ 99us) write_op.cc:270] Start()
0114 02:54:19.106596 (+ 73us) write_op.cc:276] Timestamp: P: 1736823259106445 usec, L: 0
0114 02:54:19.106610 (+ 14us) op_driver.cc:348] REPLICATION: starting
0114 02:54:19.107047 (+ 437us) log.cc:844] Serialized 356 byte log entry
0114 02:54:19.118597 (+ 11550us) op_driver.cc:464] REPLICATION: finished
0114 02:54:19.118878 (+ 281us) write_op.cc:301] APPLY: starting
0114 02:54:19.118931 (+ 53us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:19.119014 (+ 83us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:19.119104 (+ 90us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:19.119267 (+ 163us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:19.119391 (+ 124us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:19.119431 (+ 40us) write_op.cc:312] APPLY: finished
0114 02:54:19.119580 (+ 149us) log.cc:844] Serialized 23 byte log entry
0114 02:54:19.119706 (+ 126us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:19.119760 (+ 54us) write_op.cc:454] Released schema lock
0114 02:54:19.119790 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:19.119986 (+ 196us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:19.120010 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":28035,"reported_tablets":22,"spinlock_wait_cycles":10539264,"child_traces":[["op",{"apply.queue_time_us":143,"apply.run_cpu_time_us":1067,"apply.run_wall_time_us":1067,"num_ops":1,"prepare.queue_time_us":108,"prepare.run_cpu_time_us":1755,"prepare.run_wall_time_us":1754,"raft.queue_time_us":2005,"raft.run_cpu_time_us":514,"raft.run_wall_time_us":515,"replication_time_us":11910}]]}
I20250114 02:54:19.260097 463 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_1ebcd922bb294485b52278a74830aaab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:19.265497 795 tablet_service.cc:1514] Processing DeleteTablet for tablet 99b8acb1fe594bf3825c7b9121018db6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:16 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:19.266388 469 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_24cfc302131c4035b4c8e0fdc03625fa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:19.287652 1730 raft_consensus.cc:491] T 26328f8e32bc49ed9b224f0e08b9f949 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 02:54:19.147017 462 rpcz_store.cc:269] 0114 02:54:16.546045 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:16.561105 (+ 15060us) service_pool.cc:225] Handling call
0114 02:54:16.616965 (+ 55860us) spinlock_profiling.cc:243] Waited 52.7 ms on lock 0x7b54000316d4. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616cad50 00007f646174df2a 00007f64633f2dec 00007f64634eadee 00007f6461248fc4 00007f6461248f52 00007f6461248e81 00007f6461248dfc 00007f64612480b2 00007f645f512dfc 00007f645f512236
0114 02:54:19.146583 (+2529618us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:17.093026 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:17.093437 (+ 411us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:17.093467 (+ 30us) write_op.cc:435] Acquired schema lock
0114 02:54:17.093486 (+ 19us) tablet.cc:659] Decoding operations
0114 02:54:17.093771 (+ 285us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:17.093812 (+ 41us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:17.093842 (+ 30us) tablet.cc:682] Acquiring locks for 1 operations
0114 02:54:17.094930 (+ 1088us) tablet.cc:698] Row locks acquired
0114 02:54:17.094948 (+ 18us) write_op.cc:260] PREPARE: finished
0114 02:54:17.095055 (+ 107us) write_op.cc:270] Start()
0114 02:54:17.095136 (+ 81us) write_op.cc:276] Timestamp: P: 1736823257095021 usec, L: 0
0114 02:54:17.095151 (+ 15us) op_driver.cc:348] REPLICATION: starting
0114 02:54:17.095636 (+ 485us) log.cc:844] Serialized 530 byte log entry
0114 02:54:17.162247 (+ 66611us) op_driver.cc:464] REPLICATION: finished
0114 02:54:17.162929 (+ 682us) write_op.cc:301] APPLY: starting
0114 02:54:17.162986 (+ 57us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:17.163071 (+ 85us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:17.163152 (+ 81us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:17.163328 (+ 176us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:17.163454 (+ 126us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:17.163492 (+ 38us) write_op.cc:312] APPLY: finished
0114 02:54:17.163656 (+ 164us) log.cc:844] Serialized 23 byte log entry
0114 02:54:17.163783 (+ 127us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:17.163840 (+ 57us) write_op.cc:454] Released schema lock
0114 02:54:17.163868 (+ 28us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:17.164056 (+ 188us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:17.164079 (+ 23us) write_op.cc:454] Released schema lock
Related trace 'op':
0114 02:54:17.621949 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:17.622277 (+ 328us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:17.622297 (+ 20us) write_op.cc:435] Acquired schema lock
0114 02:54:17.622308 (+ 11us) tablet.cc:659] Decoding operations
0114 02:54:17.622537 (+ 229us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:17.622570 (+ 33us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:17.622590 (+ 20us) tablet.cc:682] Acquiring locks for 1 operations
0114 02:54:17.622714 (+ 124us) tablet.cc:698] Row locks acquired
0114 02:54:17.622726 (+ 12us) write_op.cc:260] PREPARE: finished
0114 02:54:17.622825 (+ 99us) write_op.cc:270] Start()
0114 02:54:17.622901 (+ 76us) write_op.cc:276] Timestamp: P: 1736823257622788 usec, L: 0
0114 02:54:17.622915 (+ 14us) op_driver.cc:348] REPLICATION: starting
0114 02:54:17.623379 (+ 464us) log.cc:844] Serialized 356 byte log entry
0114 02:54:17.891745 (+268366us) op_driver.cc:464] REPLICATION: finished
0114 02:54:17.893582 (+ 1837us) write_op.cc:301] APPLY: starting
0114 02:54:17.893621 (+ 39us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:17.893699 (+ 78us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:17.893776 (+ 77us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:17.893891 (+ 115us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:17.894005 (+ 114us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:17.894043 (+ 38us) write_op.cc:312] APPLY: finished
0114 02:54:17.894167 (+ 124us) log.cc:844] Serialized 23 byte log entry
0114 02:54:17.894269 (+ 102us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:17.894312 (+ 43us) write_op.cc:454] Released schema lock
0114 02:54:17.894337 (+ 25us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":19,"spinlock_wait_cycles":121319808,"child_traces":[["op",{"apply.queue_time_us":544,"apply.run_cpu_time_us":1100,"apply.run_wall_time_us":1099,"num_ops":1,"prepare.queue_time_us":98799,"prepare.run_cpu_time_us":1928,"prepare.run_wall_time_us":3539,"raft.queue_time_us":146,"raft.run_cpu_time_us":11,"raft.run_wall_time_us":11,"replication_time_us":67002,"spinlock_wait_cycles":141056}],["op",{"apply.queue_time_us":1722,"apply.run_cpu_time_us":900,"apply.run_wall_time_us":899,"num_ops":1,"prepare.queue_time_us":278074,"prepare.run_cpu_time_us":1609,"prepare.run_wall_time_us":2526,"replication_time_us":268752,"thread_start_us":89789,"threads_started":1}]]}
I20250114 02:54:19.304100 898 leader_election.cc:304] T 961b6a25a13a4fa2a732a6383dcd8c84 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 525198593c8e47afaee54e74267b87e3; no voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:19.305106 1730 raft_consensus.cc:513] T 26328f8e32bc49ed9b224f0e08b9f949 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:19.312085 1618 raft_consensus.cc:3054] T 961b6a25a13a4fa2a732a6383dcd8c84 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:19.314989 938 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ecf782bb3d54528aa996abca1ea3a15 with delete_type TABLET_DATA_DELETED (replaced by tablet 66bfa92db9874750b3c340e31a06c412) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:19.306525 942 tablet_service.cc:1514] Processing DeleteTablet for tablet f0bd9bd5a17d418d8082ed0ebd1bf4cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:19 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:19.308964 935 tablet_service.cc:1514] Processing DeleteTablet for tablet 290ad07ab8be4791a39194909e000843 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:13 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:19.309428 939 tablet_service.cc:1514] Processing DeleteTablet for tablet 9589afd0094545c1801b7a0b5eb8698f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:13 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:19.309957 936 tablet_service.cc:1514] Processing DeleteTablet for tablet 276942e5e54246c79f0d4f08297f58ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:14 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:19.298975 1206 tablet_bootstrap.cc:492] T 266da813b4514329be0ad97d07b7114f P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:19.318316 1206 ts_tablet_manager.cc:1397] T 266da813b4514329be0ad97d07b7114f P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.129s user 0.008s sys 0.003s
I20250114 02:54:19.319162 940 tablet_service.cc:1514] Processing DeleteTablet for tablet ba215eb8422d4f879dd0bf5996543962 with delete_type TABLET_DATA_DELETED (replaced by tablet f40c66314342453b898da3ac0e7e86bf) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:19.320436 1206 raft_consensus.cc:357] T 266da813b4514329be0ad97d07b7114f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:19.321024 1206 raft_consensus.cc:383] T 266da813b4514329be0ad97d07b7114f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:19.321255 1206 raft_consensus.cc:738] T 266da813b4514329be0ad97d07b7114f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:19.321456 1099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26328f8e32bc49ed9b224f0e08b9f949" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:19.321810 1206 consensus_queue.cc:260] T 266da813b4514329be0ad97d07b7114f P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
W20250114 02:54:19.322685 753 leader_election.cc:343] T 26328f8e32bc49ed9b224f0e08b9f949 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:19.323119 1206 ts_tablet_manager.cc:1428] T 266da813b4514329be0ad97d07b7114f P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
W20250114 02:54:19.325847 469 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_24cfc302131c4035b4c8e0fdc03625fa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250114 02:54:19.325517 463 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1ebcd922bb294485b52278a74830aaab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:19.328008 1595 ts_tablet_manager.cc:1918] T 96b06fe4173343dabd726d253b9f9b5f P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:19.328410 1595 log.cc:1198] T 96b06fe4173343dabd726d253b9f9b5f P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/96b06fe4173343dabd726d253b9f9b5f
W20250114 02:54:19.310215 470 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1ec80cfdcee64e98beccc30fec0afff8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:19.303802 1594 ts_tablet_manager.cc:1918] T ed2d137579b8467da5eed4ea2f619a9a P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:19.336768 1594 log.cc:1198] T ed2d137579b8467da5eed4ea2f619a9a P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/ed2d137579b8467da5eed4ea2f619a9a
I20250114 02:54:19.337579 1594 ts_tablet_manager.cc:1939] T ed2d137579b8467da5eed4ea2f619a9a P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:19.339447 1594 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 120) took 1864 ms. Trace:
W20250114 02:54:19.289655 751 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.66:44037, user_credentials={real_user=slave}} blocked reactor thread for 212608us
I20250114 02:54:19.346021 941 tablet_service.cc:1514] Processing DeleteTablet for tablet ed2d137579b8467da5eed4ea2f619a9a with delete_type TABLET_DATA_DELETED (replaced by tablet 0304df55431e42e1a999ed17dc35e6c0) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:19.347630 941 tablet_service.cc:1514] Processing DeleteTablet for tablet 268a12df225e4ba59d4a84c2869d4661 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:10 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:19.352406 1206 tablet_bootstrap.cc:492] T bc46815b4803418c95ea9c039a4fae88 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:19.351929 959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26328f8e32bc49ed9b224f0e08b9f949" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:19.339752 1594 rpcz_store.cc:276] 0114 02:54:17.475268 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:17.475557 (+ 289us) service_pool.cc:225] Handling call
0114 02:54:19.138179 (+1662622us) ts_tablet_manager.cc:1145] Deleting tablet ed2d137579b8467da5eed4ea2f619a9a
0114 02:54:19.138227 (+ 48us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:19.235348 (+ 97121us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:19.303552 (+ 68204us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:19.339390 (+ 35838us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":104,"tablet-delete.queue_time_us":1651130}
I20250114 02:54:19.359180 1730 leader_election.cc:290] T 26328f8e32bc49ed9b224f0e08b9f949 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:19.360352 1595 ts_tablet_manager.cc:1939] T 96b06fe4173343dabd726d253b9f9b5f P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
I20250114 02:54:19.362246 1595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 119) took 3334 ms. Trace:
I20250114 02:54:19.362716 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet ed2d137579b8467da5eed4ea2f619a9a (table default.Table_4c3843233dda407395ab9652d5870bc6 [id=0d173fdb502f482fb86b6d3e5fdc66f1]) successfully deleted
I20250114 02:54:19.374938 941 tablet_service.cc:1514] Processing DeleteTablet for tablet b63dd90f60a04465b948171e74888e79 with delete_type TABLET_DATA_DELETED (replaced by tablet 59a46bc865cc48df8e98598764fd1b54) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:19.376595 1594 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 121) took 1863 ms. Trace:
I20250114 02:54:19.392307 1595 rpcz_store.cc:276] 0114 02:54:16.027959 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:16.036871 (+ 8912us) service_pool.cc:225] Handling call
0114 02:54:19.249675 (+3212804us) ts_tablet_manager.cc:1145] Deleting tablet 96b06fe4173343dabd726d253b9f9b5f
0114 02:54:19.249719 (+ 44us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:19.297323 (+ 47604us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:19.327793 (+ 30470us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:19.362186 (+ 34393us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3157279}
I20250114 02:54:19.384744 1594 rpcz_store.cc:276] 0114 02:54:17.509975 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:17.511897 (+ 1922us) service_pool.cc:225] Handling call
0114 02:54:19.365216 (+1853319us) ts_tablet_manager.cc:1145] Deleting tablet 268a12df225e4ba59d4a84c2869d4661
0114 02:54:19.366435 (+ 1219us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:19.373035 (+ 6600us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1824124}
W20250114 02:54:19.399145 751 leader_election.cc:343] T 26328f8e32bc49ed9b224f0e08b9f949 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:19.399363 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet 96b06fe4173343dabd726d253b9f9b5f (table default.Table_d1de82caa5b94791b6c7dfed1c376e8a [id=03c775a54e9e443cbc4fab853bd6d49c]) successfully deleted
W20250114 02:54:19.400511 458 catalog_manager.cc:4839] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet 268a12df225e4ba59d4a84c2869d4661 because the tablet was not found. No further retry: Not found: Tablet not found: 268a12df225e4ba59d4a84c2869d4661
I20250114 02:54:19.401737 1595 tablet_replica.cc:331] T b63dd90f60a04465b948171e74888e79 P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:19.402350 751 leader_election.cc:304] T 26328f8e32bc49ed9b224f0e08b9f949 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:19.404398 1741 raft_consensus.cc:2743] T 26328f8e32bc49ed9b224f0e08b9f949 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:19.404884 1595 raft_consensus.cc:2238] T b63dd90f60a04465b948171e74888e79 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:19.409426 1741 raft_consensus.cc:491] T ba215eb8422d4f879dd0bf5996543962 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:19.411286 1595 raft_consensus.cc:2267] T b63dd90f60a04465b948171e74888e79 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:19.407058 1594 tablet_replica.cc:331] T c20aca761313498193e1d79c10705f8f P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:19.422724 1595 ts_tablet_manager.cc:1905] T b63dd90f60a04465b948171e74888e79 P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:19.426963 941 tablet_service.cc:1514] Processing DeleteTablet for tablet c20aca761313498193e1d79c10705f8f with delete_type TABLET_DATA_DELETED (replaced by tablet bc46815b4803418c95ea9c039a4fae88) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:19.428133 941 tablet_service.cc:1514] Processing DeleteTablet for tablet 96b06fe4173343dabd726d253b9f9b5f with delete_type TABLET_DATA_DELETED (replaced by tablet e15e6e52d7294aeeb4463b3a76022ec3) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:19.429427 941 tablet_service.cc:1514] Processing DeleteTablet for tablet db24bf3db61941ba9b5e0662db23e748 with delete_type TABLET_DATA_DELETED (replaced by tablet ddfbab181b1f474192c664cd0316037f) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:19.430755 1594 raft_consensus.cc:2238] T c20aca761313498193e1d79c10705f8f P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:19.431252 1594 raft_consensus.cc:2267] T c20aca761313498193e1d79c10705f8f P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:19.433020 1594 ts_tablet_manager.cc:1905] T c20aca761313498193e1d79c10705f8f P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:19.434448 1205 tablet_bootstrap.cc:492] T aad96f2507974023aaa988e6950feb7a P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:19.416138 1741 raft_consensus.cc:513] T ba215eb8422d4f879dd0bf5996543962 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:19.439007 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba215eb8422d4f879dd0bf5996543962" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:19.439623 962 raft_consensus.cc:2463] T ba215eb8422d4f879dd0bf5996543962 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 0.
I20250114 02:54:19.440847 751 leader_election.cc:304] T ba215eb8422d4f879dd0bf5996543962 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
I20250114 02:54:19.441680 1730 raft_consensus.cc:2798] T ba215eb8422d4f879dd0bf5996543962 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:19.442013 1730 raft_consensus.cc:491] T ba215eb8422d4f879dd0bf5996543962 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:19.442323 1730 raft_consensus.cc:3054] T ba215eb8422d4f879dd0bf5996543962 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:19.449079 941 tablet_service.cc:1514] Processing DeleteTablet for tablet 99b8acb1fe594bf3825c7b9121018db6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:16 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:19.446982 1206 tablet_bootstrap.cc:654] T bc46815b4803418c95ea9c039a4fae88 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:19.450472 941 tablet_service.cc:1514] Processing DeleteTablet for tablet 961b6a25a13a4fa2a732a6383dcd8c84 with delete_type TABLET_DATA_DELETED (replaced by tablet 99b940d872ce4781907517c40fe08781) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:19.415841 1743 tablet.cc:1721] T de4ca83a69704a368a25954cdc1b9392 P 90d6c2c58bb54e0eafb365625ac33871: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250114 02:54:19.450951 1507 raft_consensus.cc:491] T f464eabfbc0945fa8a5583f00c1dab40 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:19.449541 1730 raft_consensus.cc:513] T ba215eb8422d4f879dd0bf5996543962 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:19.451463 1507 raft_consensus.cc:513] T f464eabfbc0945fa8a5583f00c1dab40 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:19.453610 1507 leader_election.cc:290] T f464eabfbc0945fa8a5583f00c1dab40 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:19.454897 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f464eabfbc0945fa8a5583f00c1dab40" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:19.455016 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f464eabfbc0945fa8a5583f00c1dab40" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:19.455601 818 raft_consensus.cc:2388] T f464eabfbc0945fa8a5583f00c1dab40 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 90d6c2c58bb54e0eafb365625ac33871 in current term 1: Already voted for candidate 525198593c8e47afaee54e74267b87e3 in this term.
I20250114 02:54:19.456602 1042 leader_election.cc:304] T f464eabfbc0945fa8a5583f00c1dab40 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90d6c2c58bb54e0eafb365625ac33871; no voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2
I20250114 02:54:19.457620 1507 raft_consensus.cc:3054] T f464eabfbc0945fa8a5583f00c1dab40 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:19.457692 1205 ts_tablet_manager.cc:1397] T aad96f2507974023aaa988e6950feb7a P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.307s user 0.008s sys 0.002s
I20250114 02:54:19.469935 1507 raft_consensus.cc:2743] T f464eabfbc0945fa8a5583f00c1dab40 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:19.479444 1207 tablet_bootstrap.cc:654] T ba215eb8422d4f879dd0bf5996543962 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:19.480592 1101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba215eb8422d4f879dd0bf5996543962" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
W20250114 02:54:19.482491 753 leader_election.cc:343] T ba215eb8422d4f879dd0bf5996543962 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:19.490738 1205 raft_consensus.cc:357] T aad96f2507974023aaa988e6950feb7a P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:19.491438 1205 raft_consensus.cc:383] T aad96f2507974023aaa988e6950feb7a P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:19.491739 1205 raft_consensus.cc:738] T aad96f2507974023aaa988e6950feb7a P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:19.493438 1741 leader_election.cc:290] T ba215eb8422d4f879dd0bf5996543962 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:19.495605 1099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba215eb8422d4f879dd0bf5996543962" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
W20250114 02:54:19.497321 753 leader_election.cc:343] T ba215eb8422d4f879dd0bf5996543962 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:19.495914 1730 leader_election.cc:290] T ba215eb8422d4f879dd0bf5996543962 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Requested vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:19.509740 1730 raft_consensus.cc:491] T 321b9ad6d1ac4d1c80c7a01faf2015af P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:19.510334 1730 raft_consensus.cc:513] T 321b9ad6d1ac4d1c80c7a01faf2015af P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:19.515266 1099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321b9ad6d1ac4d1c80c7a01faf2015af" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
W20250114 02:54:19.517019 753 leader_election.cc:343] T 321b9ad6d1ac4d1c80c7a01faf2015af P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:19.523514 959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba215eb8422d4f879dd0bf5996543962" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
I20250114 02:54:19.524081 959 raft_consensus.cc:3054] T ba215eb8422d4f879dd0bf5996543962 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:19.524617 1595 ts_tablet_manager.cc:1918] T b63dd90f60a04465b948171e74888e79 P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250114 02:54:19.527407 1749 tablet.cc:1721] T a3fdab330d574c0283ac58bc9e450b87 P 525198593c8e47afaee54e74267b87e3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250114 02:54:19.528640 1595 log.cc:1198] T b63dd90f60a04465b948171e74888e79 P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/b63dd90f60a04465b948171e74888e79
I20250114 02:54:19.529556 1595 ts_tablet_manager.cc:1939] T b63dd90f60a04465b948171e74888e79 P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
I20250114 02:54:19.529783 1750 tablet.cc:1721] T a3fdab330d574c0283ac58bc9e450b87 P 90d6c2c58bb54e0eafb365625ac33871: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250114 02:54:19.531872 1595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 126) took 2635 ms. Trace:
I20250114 02:54:19.534112 1206 tablet_bootstrap.cc:492] T bc46815b4803418c95ea9c039a4fae88 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:19.534590 1206 ts_tablet_manager.cc:1397] T bc46815b4803418c95ea9c039a4fae88 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.211s user 0.008s sys 0.007s
I20250114 02:54:19.534284 1595 rpcz_store.cc:276] 0114 02:54:16.896355 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:16.970405 (+ 74050us) service_pool.cc:225] Handling call
0114 02:54:19.401581 (+2431176us) ts_tablet_manager.cc:1145] Deleting tablet b63dd90f60a04465b948171e74888e79
0114 02:54:19.401625 (+ 44us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:19.462657 (+ 61032us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:19.524197 (+ 61540us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:19.531818 (+ 7621us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9389,"tablet-delete.queue_time_us":2430471}
I20250114 02:54:19.536331 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet b63dd90f60a04465b948171e74888e79 (table default.Table_1097f09fb10e441fb639f8d7aa12d487 [id=d8c9691818d042d7b49475fa009c4ca1]) successfully deleted
I20250114 02:54:19.536449 1730 leader_election.cc:290] T 321b9ad6d1ac4d1c80c7a01faf2015af P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:19.537135 1206 raft_consensus.cc:357] T bc46815b4803418c95ea9c039a4fae88 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:19.537982 1206 raft_consensus.cc:383] T bc46815b4803418c95ea9c039a4fae88 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:19.538268 1206 raft_consensus.cc:738] T bc46815b4803418c95ea9c039a4fae88 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:19.538403 1595 tablet_replica.cc:331] T 99b8acb1fe594bf3825c7b9121018db6 P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:19.539321 1595 raft_consensus.cc:2238] T 99b8acb1fe594bf3825c7b9121018db6 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:19.538940 1206 consensus_queue.cc:260] T bc46815b4803418c95ea9c039a4fae88 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:19.545594 469 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_2217d3b1dcf4455d9103cad045518573" } modify_external_catalogs: true
I20250114 02:54:19.546092 469 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_2217d3b1dcf4455d9103cad045518573" } modify_external_catalogs: true
I20250114 02:54:19.546730 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321b9ad6d1ac4d1c80c7a01faf2015af" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:19.556265 959 raft_consensus.cc:2463] T ba215eb8422d4f879dd0bf5996543962 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 1.
W20250114 02:54:19.562114 751 leader_election.cc:343] T 321b9ad6d1ac4d1c80c7a01faf2015af P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:19.562530 751 leader_election.cc:304] T 321b9ad6d1ac4d1c80c7a01faf2015af P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:19.563884 1741 raft_consensus.cc:2743] T 321b9ad6d1ac4d1c80c7a01faf2015af P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:19.563998 751 leader_election.cc:304] T ba215eb8422d4f879dd0bf5996543962 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters: 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:19.565047 1730 raft_consensus.cc:2798] T ba215eb8422d4f879dd0bf5996543962 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:19.565495 1730 raft_consensus.cc:695] T ba215eb8422d4f879dd0bf5996543962 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 LEADER]: Becoming Leader. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Running, Role: LEADER
I20250114 02:54:19.566187 1730 consensus_queue.cc:237] T ba215eb8422d4f879dd0bf5996543962 P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:19.583616 463 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_ae220acb7e0345bdbb80b11398e4d153" } modify_external_catalogs: true
I20250114 02:54:19.584182 463 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_ae220acb7e0345bdbb80b11398e4d153" } modify_external_catalogs: true
I20250114 02:54:19.584421 1595 raft_consensus.cc:2267] T 99b8acb1fe594bf3825c7b9121018db6 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:19.586097 1595 ts_tablet_manager.cc:1905] T 99b8acb1fe594bf3825c7b9121018db6 P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:19.614785 1702 ts_tablet_manager.cc:1918] T 00080bea6c8e46e584ccba6dcb0e221b P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:19.615197 1702 log.cc:1198] T 00080bea6c8e46e584ccba6dcb0e221b P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/00080bea6c8e46e584ccba6dcb0e221b
W20250114 02:54:19.617036 468 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 044832851f804208b0a371eb0857dbf3 consensus state has no leader
I20250114 02:54:19.617882 468 master_service.cc:946] ReplaceTablet: received request to replace tablet ca986f81ed6848a89588366f3f400eec from {username='slave'} at 127.0.0.1:33540
W20250114 02:54:19.625766 464 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ce7d6088fc884a248918d2348adcaccf consensus state has no leader
I20250114 02:54:19.635506 1741 raft_consensus.cc:491] T bc46815b4803418c95ea9c039a4fae88 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:19.636051 1741 raft_consensus.cc:513] T bc46815b4803418c95ea9c039a4fae88 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:19.638176 1741 leader_election.cc:290] T bc46815b4803418c95ea9c039a4fae88 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:19.639142 1101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc46815b4803418c95ea9c039a4fae88" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:19.639060 1733 tablet.cc:1721] T a3fdab330d574c0283ac58bc9e450b87 P 819f9e3412e54d44abb11d8b3ca877c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250114 02:54:19.639861 959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc46815b4803418c95ea9c039a4fae88" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
W20250114 02:54:19.640717 753 leader_election.cc:343] T bc46815b4803418c95ea9c039a4fae88 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
W20250114 02:54:19.643041 751 leader_election.cc:343] T bc46815b4803418c95ea9c039a4fae88 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:19.643421 751 leader_election.cc:304] T bc46815b4803418c95ea9c039a4fae88 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:19.644292 1652 raft_consensus.cc:2743] T bc46815b4803418c95ea9c039a4fae88 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:19.646046 1206 ts_tablet_manager.cc:1428] T bc46815b4803418c95ea9c039a4fae88 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.111s user 0.008s sys 0.000s
I20250114 02:54:19.646854 1206 tablet_bootstrap.cc:492] T 53a8bb487ad64157ad3baecf5d866eca P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:19.487572 1618 raft_consensus.cc:2743] T 961b6a25a13a4fa2a732a6383dcd8c84 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:19.661484 1702 ts_tablet_manager.cc:1939] T 00080bea6c8e46e584ccba6dcb0e221b P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:19.667279 1205 consensus_queue.cc:260] T aad96f2507974023aaa988e6950feb7a P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:19.669245 1205 ts_tablet_manager.cc:1428] T aad96f2507974023aaa988e6950feb7a P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.211s user 0.005s sys 0.000s
I20250114 02:54:19.677702 1594 ts_tablet_manager.cc:1918] T c20aca761313498193e1d79c10705f8f P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:19.678052 1594 log.cc:1198] T c20aca761313498193e1d79c10705f8f P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/c20aca761313498193e1d79c10705f8f
I20250114 02:54:19.680086 1618 raft_consensus.cc:491] T aad96f2507974023aaa988e6950feb7a P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:19.681195 1618 raft_consensus.cc:513] T aad96f2507974023aaa988e6950feb7a P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:19.682667 1702 tablet_replica.cc:331] T 7ecf782bb3d54528aa996abca1ea3a15 P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:19.683673 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aad96f2507974023aaa988e6950feb7a" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:19.685490 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet 00080bea6c8e46e584ccba6dcb0e221b (table default.Table_eb7878bdce82445c97f6ce665eb52d8e [id=3adecb85db3b44a4a65e50e61a91e908]) successfully deleted
I20250114 02:54:19.687103 1595 ts_tablet_manager.cc:1918] T 99b8acb1fe594bf3825c7b9121018db6 P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:19.691762 1594 ts_tablet_manager.cc:1939] T c20aca761313498193e1d79c10705f8f P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:19.693950 1618 leader_election.cc:290] T aad96f2507974023aaa988e6950feb7a P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:19.694128 1594 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 122) took 2105 ms. Trace:
I20250114 02:54:19.693476 1101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aad96f2507974023aaa988e6950feb7a" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:19.694839 1101 raft_consensus.cc:2388] T aad96f2507974023aaa988e6950feb7a P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 525198593c8e47afaee54e74267b87e3 in current term 1: Already voted for candidate 819f9e3412e54d44abb11d8b3ca877c2 in this term.
I20250114 02:54:19.694470 1594 rpcz_store.cc:276] 0114 02:54:17.588612 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:17.595664 (+ 7052us) service_pool.cc:225] Handling call
0114 02:54:19.403911 (+1808247us) ts_tablet_manager.cc:1145] Deleting tablet c20aca761313498193e1d79c10705f8f
0114 02:54:19.404803 (+ 892us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:19.530369 (+125566us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:19.624114 (+ 93745us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:19.694062 (+ 69948us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1806690}
I20250114 02:54:19.695552 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet c20aca761313498193e1d79c10705f8f (table default.Table_6a4fe39298514187b65def5841d3a992 [id=5c5a7974ad2f4e58ba5c47a6eeaaac62]) successfully deleted
I20250114 02:54:19.695814 898 leader_election.cc:304] T aad96f2507974023aaa988e6950feb7a P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 525198593c8e47afaee54e74267b87e3; no voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:19.696602 1595 log.cc:1198] T 99b8acb1fe594bf3825c7b9121018db6 P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/99b8acb1fe594bf3825c7b9121018db6
I20250114 02:54:19.697288 1618 raft_consensus.cc:3054] T aad96f2507974023aaa988e6950feb7a P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:19.698028 1595 ts_tablet_manager.cc:1939] T 99b8acb1fe594bf3825c7b9121018db6 P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
W20250114 02:54:19.699858 455 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 044832851f804208b0a371eb0857dbf3 (table default.Table_0518d8bed7304c86aff32e05e6f07c0d [id=515a2fe3f1e84e82bccc02ef5cecef1b], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 044832851f804208b0a371eb0857dbf3 consensus state has no leader
I20250114 02:54:19.700528 1206 tablet_bootstrap.cc:654] T 53a8bb487ad64157ad3baecf5d866eca P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:19.707105 1207 tablet_bootstrap.cc:492] T ba215eb8422d4f879dd0bf5996543962 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:19.707494 1207 ts_tablet_manager.cc:1397] T ba215eb8422d4f879dd0bf5996543962 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.472s user 0.006s sys 0.006s
I20250114 02:54:19.712114 1205 tablet_bootstrap.cc:492] T 321b9ad6d1ac4d1c80c7a01faf2015af P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:19.684016 1702 raft_consensus.cc:2238] T 7ecf782bb3d54528aa996abca1ea3a15 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:19.717274 1597 consensus_queue.cc:1035] T f464eabfbc0945fa8a5583f00c1dab40 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:19.717274 1702 raft_consensus.cc:2267] T 7ecf782bb3d54528aa996abca1ea3a15 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:19.729208 462 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 55b68586ef8b45439a7dd8dcd2efcd79 (table default.Table_66f2263162f64250a0b49073abe4bd4c [id=c73ede1a5f664d8eb204fb9a012f8f7a]). Expected version 1 got 0
I20250114 02:54:19.729379 1081 tablet_service.cc:1467] Processing CreateTablet for tablet a52d6749ff834b62a8e20a2f45935240 (DEFAULT_TABLE table=default.Table_fee886c2440a47bfa5a5875d58ef45ad [id=b3ee15b9f10f4d489edbde3af9034d25]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:19.730284 1081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a52d6749ff834b62a8e20a2f45935240. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:19.729852 462 catalog_manager.cc:5526] T aad96f2507974023aaa988e6950feb7a P 819f9e3412e54d44abb11d8b3ca877c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65). New cstate: current_term: 1 leader_uuid: "819f9e3412e54d44abb11d8b3ca877c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:19.730926 462 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet aad96f2507974023aaa988e6950feb7a (table default.Table_9676926a50eb42349a2d609b30566e9a [id=f38063ebf94d4084ab0174428503dc29]). Expected version 2 got 0
I20250114 02:54:19.732843 795 tablet_service.cc:1467] Processing CreateTablet for tablet a52d6749ff834b62a8e20a2f45935240 (DEFAULT_TABLE table=default.Table_fee886c2440a47bfa5a5875d58ef45ad [id=b3ee15b9f10f4d489edbde3af9034d25]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:19.733343 1082 tablet_service.cc:1467] Processing CreateTablet for tablet 32dd43e8a3544dbaa6e2fbb4b4c1dc28 (DEFAULT_TABLE table=default.Table_693a2c1631b347af8d9d873d6b08eb92 [id=ef8b70ec66f84e2182a5728ae653b8a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:19.734026 795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a52d6749ff834b62a8e20a2f45935240. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:19.733929 941 tablet_service.cc:1467] Processing CreateTablet for tablet a52d6749ff834b62a8e20a2f45935240 (DEFAULT_TABLE table=default.Table_fee886c2440a47bfa5a5875d58ef45ad [id=b3ee15b9f10f4d489edbde3af9034d25]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:19.734892 941 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a52d6749ff834b62a8e20a2f45935240. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:19.734937 1082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32dd43e8a3544dbaa6e2fbb4b4c1dc28. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:19.736908 792 tablet_service.cc:1467] Processing CreateTablet for tablet 32dd43e8a3544dbaa6e2fbb4b4c1dc28 (DEFAULT_TABLE table=default.Table_693a2c1631b347af8d9d873d6b08eb92 [id=ef8b70ec66f84e2182a5728ae653b8a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:19.737067 1083 tablet_service.cc:1467] Processing CreateTablet for tablet 6e858a8449444ef999b9bbef612249d3 (DEFAULT_TABLE table=default.Table_8be12474c1844678b8d3847682d60ee5 [id=a17ad30ef32f4cbcb22b5a7d1c7096ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:19.737794 792 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32dd43e8a3544dbaa6e2fbb4b4c1dc28. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:19.737939 1083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e858a8449444ef999b9bbef612249d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:19.740410 1080 tablet_service.cc:1467] Processing CreateTablet for tablet 38fe8829d98547fb907ef78099df3d7f (DEFAULT_TABLE table=default.Table_60450873174a4c80aa7e2122343d82be [id=9299683349354f879847b7c9d93f6072]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:19.740991 791 tablet_service.cc:1467] Processing CreateTablet for tablet 6e858a8449444ef999b9bbef612249d3 (DEFAULT_TABLE table=default.Table_8be12474c1844678b8d3847682d60ee5 [id=a17ad30ef32f4cbcb22b5a7d1c7096ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:19.741359 1080 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38fe8829d98547fb907ef78099df3d7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:19.741690 1594 tablet_replica.cc:331] T a3886c5f976445cebc95ef1839a74aa9 P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:19.742131 798 tablet_service.cc:1467] Processing CreateTablet for tablet 38fe8829d98547fb907ef78099df3d7f (DEFAULT_TABLE table=default.Table_60450873174a4c80aa7e2122343d82be [id=9299683349354f879847b7c9d93f6072]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:19.742493 1594 raft_consensus.cc:2238] T a3886c5f976445cebc95ef1839a74aa9 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 LEADER]: Raft consensus shutting down.
I20250114 02:54:19.742941 798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38fe8829d98547fb907ef78099df3d7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:19.743271 1594 raft_consensus.cc:2267] T a3886c5f976445cebc95ef1839a74aa9 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:19.744716 791 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e858a8449444ef999b9bbef612249d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:19.745167 940 tablet_service.cc:1467] Processing CreateTablet for tablet 32dd43e8a3544dbaa6e2fbb4b4c1dc28 (DEFAULT_TABLE table=default.Table_693a2c1631b347af8d9d873d6b08eb92 [id=ef8b70ec66f84e2182a5728ae653b8a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:19.746019 940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32dd43e8a3544dbaa6e2fbb4b4c1dc28. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:19.753748 466 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_cec05793ad284a18a7da3557696f5bcb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:19.755214 466 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cec05793ad284a18a7da3557696f5bcb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:19.766376 942 tablet_service.cc:1467] Processing CreateTablet for tablet 6e858a8449444ef999b9bbef612249d3 (DEFAULT_TABLE table=default.Table_8be12474c1844678b8d3847682d60ee5 [id=a17ad30ef32f4cbcb22b5a7d1c7096ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:19.767453 942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e858a8449444ef999b9bbef612249d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:19.772327 463 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet 266da813b4514329be0ad97d07b7114f
I20250114 02:54:19.774019 794 tablet_service.cc:1514] Processing DeleteTablet for tablet 266da813b4514329be0ad97d07b7114f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:19 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:19.775820 1086 tablet_service.cc:1514] Processing DeleteTablet for tablet 266da813b4514329be0ad97d07b7114f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:19 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:19.777221 936 tablet_service.cc:1467] Processing CreateTablet for tablet 38fe8829d98547fb907ef78099df3d7f (DEFAULT_TABLE table=default.Table_60450873174a4c80aa7e2122343d82be [id=9299683349354f879847b7c9d93f6072]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:19.778134 936 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38fe8829d98547fb907ef78099df3d7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:19.778393 1594 ts_tablet_manager.cc:1905] T a3886c5f976445cebc95ef1839a74aa9 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:19.782951 1086 tablet_service.cc:1467] Processing CreateTablet for tablet d05089eae2b74f9fb23a226d358a59da (DEFAULT_TABLE table=default.Table_5e1c3ceefda2466da137a659035f9ae4 [id=ee635d0ed76b46c9b2e139a75296017b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:19.783825 1086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d05089eae2b74f9fb23a226d358a59da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:19.787145 1595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 127) took 2684 ms. Trace:
I20250114 02:54:19.787473 1595 rpcz_store.cc:276] 0114 02:54:17.103023 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:17.103164 (+ 141us) service_pool.cc:225] Handling call
0114 02:54:19.538099 (+2434935us) ts_tablet_manager.cc:1145] Deleting tablet 99b8acb1fe594bf3825c7b9121018db6
0114 02:54:19.538280 (+ 181us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:19.626001 (+ 87721us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:19.686854 (+ 60853us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:19.787088 (+100234us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2434409}
I20250114 02:54:19.788599 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet 99b8acb1fe594bf3825c7b9121018db6 (table default.Table_661ed49bcbbb476aa68742b72604d95f [id=00383db95e10461a83394c5bd94ca94a]) successfully deleted
I20250114 02:54:19.792210 939 tablet_service.cc:1467] Processing CreateTablet for tablet d05089eae2b74f9fb23a226d358a59da (DEFAULT_TABLE table=default.Table_5e1c3ceefda2466da137a659035f9ae4 [id=ee635d0ed76b46c9b2e139a75296017b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:19.793109 939 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d05089eae2b74f9fb23a226d358a59da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:19.808307 1206 tablet_bootstrap.cc:492] T 53a8bb487ad64157ad3baecf5d866eca P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:19.808802 1206 ts_tablet_manager.cc:1397] T 53a8bb487ad64157ad3baecf5d866eca P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.162s user 0.008s sys 0.002s
I20250114 02:54:19.810968 1206 raft_consensus.cc:357] T 53a8bb487ad64157ad3baecf5d866eca P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:19.811630 1206 raft_consensus.cc:383] T 53a8bb487ad64157ad3baecf5d866eca P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:19.811895 1206 raft_consensus.cc:738] T 53a8bb487ad64157ad3baecf5d866eca P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:19.820838 1083 tablet_service.cc:1467] Processing CreateTablet for tablet c86562688d5b4fffb71cbdf7be0afc37 (DEFAULT_TABLE table=default.Table_7176743e4429467481ed67254643b9f9 [id=22e634dc594d4962b404424977f2cd74]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:19.821785 1083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c86562688d5b4fffb71cbdf7be0afc37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:19.824151 798 tablet_service.cc:1467] Processing CreateTablet for tablet 719188f639724851a5fc2b37cb60fdbb (DEFAULT_TABLE table=default.Table_77652aa0d7a94f1f98afb881a924eb24 [id=e6292728826f437cb75c524a8aef2e87]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:19.826226 1082 tablet_service.cc:1467] Processing CreateTablet for tablet 719188f639724851a5fc2b37cb60fdbb (DEFAULT_TABLE table=default.Table_77652aa0d7a94f1f98afb881a924eb24 [id=e6292728826f437cb75c524a8aef2e87]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:19.827452 1082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 719188f639724851a5fc2b37cb60fdbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:19.709620 1207 raft_consensus.cc:357] T ba215eb8422d4f879dd0bf5996543962 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:19.820070 791 tablet_service.cc:1467] Processing CreateTablet for tablet d05089eae2b74f9fb23a226d358a59da (DEFAULT_TABLE table=default.Table_5e1c3ceefda2466da137a659035f9ae4 [id=ee635d0ed76b46c9b2e139a75296017b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:19.837532 1207 raft_consensus.cc:383] T ba215eb8422d4f879dd0bf5996543962 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:19.839818 1713 consensus_queue.cc:1035] T f464eabfbc0945fa8a5583f00c1dab40 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:54:19.841732 1206 consensus_queue.cc:260] T 53a8bb487ad64157ad3baecf5d866eca P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:19.843351 1595 tablet_replica.cc:331] T 961b6a25a13a4fa2a732a6383dcd8c84 P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:19.844116 1595 raft_consensus.cc:2238] T 961b6a25a13a4fa2a732a6383dcd8c84 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:19.845484 1206 ts_tablet_manager.cc:1428] T 53a8bb487ad64157ad3baecf5d866eca P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.036s user 0.006s sys 0.000s
I20250114 02:54:19.845682 1595 raft_consensus.cc:2267] T 961b6a25a13a4fa2a732a6383dcd8c84 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:19.846863 1206 tablet_bootstrap.cc:492] T 66bfa92db9874750b3c340e31a06c412 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:19.847846 1595 ts_tablet_manager.cc:1905] T 961b6a25a13a4fa2a732a6383dcd8c84 P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:19.855305 792 tablet_service.cc:1467] Processing CreateTablet for tablet c86562688d5b4fffb71cbdf7be0afc37 (DEFAULT_TABLE table=default.Table_7176743e4429467481ed67254643b9f9 [id=22e634dc594d4962b404424977f2cd74]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:19.858981 1618 raft_consensus.cc:2743] T aad96f2507974023aaa988e6950feb7a P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:19.860793 798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 719188f639724851a5fc2b37cb60fdbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:19.862708 791 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d05089eae2b74f9fb23a226d358a59da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:19.870785 792 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c86562688d5b4fffb71cbdf7be0afc37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:19.872399 795 tablet_service.cc:1467] Processing CreateTablet for tablet 6e2fb05f72b743e2a02ed36b63f80b23 (DEFAULT_TABLE table=default.Table_395ca40035c8410ca1ea6b0e1d54bf37 [id=4ce0efd1198d4af28694b13aab4299fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:19.873593 795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e2fb05f72b743e2a02ed36b63f80b23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:19.874159 794 tablet_service.cc:1467] Processing CreateTablet for tablet 99b940d872ce4781907517c40fe08781 (DEFAULT_TABLE table=default.Table_0943b1ca347042c9a6043006bc566701 [id=e3418787ef0a4e22979fb41d4f94ebb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:19.874460 1730 consensus_queue.cc:1035] T aad96f2507974023aaa988e6950feb7a P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:19.875087 794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99b940d872ce4781907517c40fe08781. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:19.878731 1081 tablet_service.cc:1467] Processing CreateTablet for tablet 6e2fb05f72b743e2a02ed36b63f80b23 (DEFAULT_TABLE table=default.Table_395ca40035c8410ca1ea6b0e1d54bf37 [id=4ce0efd1198d4af28694b13aab4299fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:19.879746 1081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e2fb05f72b743e2a02ed36b63f80b23. 1 dirs total, 0 dirs full, 0 dirs failed
W20250114 02:54:19.881639 898 consensus_peers.cc:487] T 99b8acb1fe594bf3825c7b9121018db6 P 525198593c8e47afaee54e74267b87e3 -> Peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Couldn't send request to peer 819f9e3412e54d44abb11d8b3ca877c2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99b8acb1fe594bf3825c7b9121018db6. This is attempt 1: this message will repeat every 5th retry.
I20250114 02:54:19.888012 935 tablet_service.cc:1467] Processing CreateTablet for tablet 719188f639724851a5fc2b37cb60fdbb (DEFAULT_TABLE table=default.Table_77652aa0d7a94f1f98afb881a924eb24 [id=e6292728826f437cb75c524a8aef2e87]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:19.880676 1080 tablet_service.cc:1467] Processing CreateTablet for tablet 99b940d872ce4781907517c40fe08781 (DEFAULT_TABLE table=default.Table_0943b1ca347042c9a6043006bc566701 [id=e3418787ef0a4e22979fb41d4f94ebb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:19.890671 938 tablet_service.cc:1467] Processing CreateTablet for tablet c86562688d5b4fffb71cbdf7be0afc37 (DEFAULT_TABLE table=default.Table_7176743e4429467481ed67254643b9f9 [id=22e634dc594d4962b404424977f2cd74]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:19.890714 937 tablet_service.cc:1467] Processing CreateTablet for tablet 6e2fb05f72b743e2a02ed36b63f80b23 (DEFAULT_TABLE table=default.Table_395ca40035c8410ca1ea6b0e1d54bf37 [id=4ce0efd1198d4af28694b13aab4299fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:19.891983 934 tablet_service.cc:1467] Processing CreateTablet for tablet 99b940d872ce4781907517c40fe08781 (DEFAULT_TABLE table=default.Table_0943b1ca347042c9a6043006bc566701 [id=e3418787ef0a4e22979fb41d4f94ebb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:19.892694 1080 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99b940d872ce4781907517c40fe08781. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:19.898945 1206 tablet_bootstrap.cc:654] T 66bfa92db9874750b3c340e31a06c412 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:19.900856 464 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_8be12474c1844678b8d3847682d60ee5" } new_table_name: "default.Table_5a9480c6474848d6b400e7dd274d15b0" modify_external_catalogs: true
I20250114 02:54:19.905000 470 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_fa5f4f55ea844a8c8af8acb80f0fd6a4" } new_table_name: "default.Table_63b25aa4a2aa4e58a1e1b6835d0b25ce" modify_external_catalogs: true
I20250114 02:54:19.905058 463 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_2850a159f211445988d540f3a56ec551"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:19.906584 463 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2850a159f211445988d540f3a56ec551 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:19.909610 1086 tablet_service.cc:1467] Processing CreateTablet for tablet fc406eab968745d893f1a8538ff61f37 (DEFAULT_TABLE table=default.Table_debb45fd7c4b4826a73d39a6a6df88c4 [id=e1ca4114e55d48839dfa2f76624a4234]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:19.910606 1086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc406eab968745d893f1a8538ff61f37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:19.912865 1084 tablet_service.cc:1467] Processing CreateTablet for tablet df045d80d8ce4ff194a6d2039a44bce6 (DEFAULT_TABLE table=default.Table_df476a4b13be4c11a896663f6211624d [id=da0258578bb34ae29551fc593bf7d0b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:19.912954 796 tablet_service.cc:1467] Processing CreateTablet for tablet df045d80d8ce4ff194a6d2039a44bce6 (DEFAULT_TABLE table=default.Table_df476a4b13be4c11a896663f6211624d [id=da0258578bb34ae29551fc593bf7d0b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:19.913769 1084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df045d80d8ce4ff194a6d2039a44bce6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:19.916464 1702 ts_tablet_manager.cc:1905] T 7ecf782bb3d54528aa996abca1ea3a15 P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:19.917101 790 tablet_service.cc:1467] Processing CreateTablet for tablet fc406eab968745d893f1a8538ff61f37 (DEFAULT_TABLE table=default.Table_debb45fd7c4b4826a73d39a6a6df88c4 [id=e1ca4114e55d48839dfa2f76624a4234]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:19.838405 1207 raft_consensus.cc:738] T ba215eb8422d4f879dd0bf5996543962 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:19.922143 793 tablet_service.cc:1467] Processing CreateTablet for tablet 59a46bc865cc48df8e98598764fd1b54 (DEFAULT_TABLE table=default.Table_1097f09fb10e441fb639f8d7aa12d487 [id=d8c9691818d042d7b49475fa009c4ca1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:19.928983 935 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 719188f639724851a5fc2b37cb60fdbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:19.940609 1507 consensus_queue.cc:1035] T fcb58a05f980460884ed81a30a5d59c2 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:19.942224 938 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c86562688d5b4fffb71cbdf7be0afc37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:19.944500 934 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99b940d872ce4781907517c40fe08781. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:19.945154 937 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e2fb05f72b743e2a02ed36b63f80b23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:19.964732 797 tablet_service.cc:1467] Processing CreateTablet for tablet ed7f575653c848d7b076c17c38c4a244 (DEFAULT_TABLE table=default.Table_44d578b4433242d5b7d8a61f6e4213e7 [id=954669573a3c4a97a488805e05b0b220]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:19.918905 1207 consensus_queue.cc:260] T ba215eb8422d4f879dd0bf5996543962 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:19.973417 1207 ts_tablet_manager.cc:1428] T ba215eb8422d4f879dd0bf5996543962 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.266s user 0.007s sys 0.000s
I20250114 02:54:19.974287 1207 tablet_bootstrap.cc:492] T 26328f8e32bc49ed9b224f0e08b9f949 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:19.974365 1083 tablet_service.cc:1467] Processing CreateTablet for tablet ed7f575653c848d7b076c17c38c4a244 (DEFAULT_TABLE table=default.Table_44d578b4433242d5b7d8a61f6e4213e7 [id=954669573a3c4a97a488805e05b0b220]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:19.981091 1082 tablet_service.cc:1467] Processing CreateTablet for tablet 97f4e061d3b94ee9982410b7ce135453 (DEFAULT_TABLE table=default.Table_6a4fe39298514187b65def5841d3a992 [id=5c5a7974ad2f4e58ba5c47a6eeaaac62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:19.982339 1507 consensus_queue.cc:1035] T fcb58a05f980460884ed81a30a5d59c2 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:20.007227 1086 tablet_service.cc:1467] Processing CreateTablet for tablet 59a46bc865cc48df8e98598764fd1b54 (DEFAULT_TABLE table=default.Table_1097f09fb10e441fb639f8d7aa12d487 [id=d8c9691818d042d7b49475fa009c4ca1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:20.013763 1205 tablet_bootstrap.cc:654] T 321b9ad6d1ac4d1c80c7a01faf2015af P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
W20250114 02:54:20.016090 898 consensus_peers.cc:487] T 99b8acb1fe594bf3825c7b9121018db6 P 525198593c8e47afaee54e74267b87e3 -> Peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Couldn't send request to peer 90d6c2c58bb54e0eafb365625ac33871. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99b8acb1fe594bf3825c7b9121018db6. This is attempt 1: this message will repeat every 5th retry.
I20250114 02:54:20.016779 936 tablet_service.cc:1514] Processing DeleteTablet for tablet 266da813b4514329be0ad97d07b7114f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:19 UTC) from {username='slave'} at 127.0.0.1:55068
W20250114 02:54:20.020537 753 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.67:45749, user_credentials={real_user=slave}} blocked reactor thread for 48957.3us
I20250114 02:54:20.019922 789 tablet_service.cc:1467] Processing CreateTablet for tablet 97f4e061d3b94ee9982410b7ce135453 (DEFAULT_TABLE table=default.Table_6a4fe39298514187b65def5841d3a992 [id=5c5a7974ad2f4e58ba5c47a6eeaaac62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:20.022459 1084 tablet_service.cc:1467] Processing CreateTablet for tablet 3161f75bc4e84000af022faa9804c810 (DEFAULT_TABLE table=default.Table_35a78676616e4b20906b262c9d0c0d9e [id=e984149e9da444f1b2802e3f35a60fd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:20.023094 788 tablet_service.cc:1467] Processing CreateTablet for tablet 3161f75bc4e84000af022faa9804c810 (DEFAULT_TABLE table=default.Table_35a78676616e4b20906b262c9d0c0d9e [id=e984149e9da444f1b2802e3f35a60fd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:20.024472 1652 raft_consensus.cc:491] T 53a8bb487ad64157ad3baecf5d866eca P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:20.024875 1652 raft_consensus.cc:513] T 53a8bb487ad64157ad3baecf5d866eca P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:20.026615 1652 leader_election.cc:290] T 53a8bb487ad64157ad3baecf5d866eca P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:20.030989 1099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53a8bb487ad64157ad3baecf5d866eca" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:20.031966 1594 ts_tablet_manager.cc:1918] T a3886c5f976445cebc95ef1839a74aa9 P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250114 02:54:20.032374 1594 log.cc:1198] T a3886c5f976445cebc95ef1839a74aa9 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/a3886c5f976445cebc95ef1839a74aa9
W20250114 02:54:20.032480 753 leader_election.cc:343] T 53a8bb487ad64157ad3baecf5d866eca P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:20.033046 1594 ts_tablet_manager.cc:1939] T a3886c5f976445cebc95ef1839a74aa9 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:20.033607 1083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed7f575653c848d7b076c17c38c4a244. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:20.034162 1084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3161f75bc4e84000af022faa9804c810. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:20.035497 1086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59a46bc865cc48df8e98598764fd1b54. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:20.038008 1082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97f4e061d3b94ee9982410b7ce135453. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:20.040614 793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59a46bc865cc48df8e98598764fd1b54. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:20.042347 1741 consensus_queue.cc:1035] T aad96f2507974023aaa988e6950feb7a P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:20.042677 790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc406eab968745d893f1a8538ff61f37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:20.044549 796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df045d80d8ce4ff194a6d2039a44bce6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:20.045939 797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed7f575653c848d7b076c17c38c4a244. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:20.047605 788 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3161f75bc4e84000af022faa9804c810. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:20.053280 1206 tablet_bootstrap.cc:492] T 66bfa92db9874750b3c340e31a06c412 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:20.053823 789 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97f4e061d3b94ee9982410b7ce135453. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:20.055117 1594 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 124) took 1686 ms. Trace:
I20250114 02:54:20.055433 1594 rpcz_store.cc:276] 0114 02:54:18.368465 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:18.368581 (+ 116us) service_pool.cc:225] Handling call
0114 02:54:19.741464 (+1372883us) ts_tablet_manager.cc:1145] Deleting tablet a3886c5f976445cebc95ef1839a74aa9
0114 02:54:19.741582 (+ 118us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:19.860665 (+119083us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:20.031751 (+171086us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:20.055059 (+ 23308us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11136,"tablet-delete.queue_time_us":1372331}
I20250114 02:54:20.056509 1594 tablet_replica.cc:331] T 53a8bb487ad64157ad3baecf5d866eca P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:20.057227 1594 raft_consensus.cc:2238] T 53a8bb487ad64157ad3baecf5d866eca P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:20.057634 1594 raft_consensus.cc:2267] T 53a8bb487ad64157ad3baecf5d866eca P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:20.059514 1594 ts_tablet_manager.cc:1905] T 53a8bb487ad64157ad3baecf5d866eca P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:20.064422 1206 ts_tablet_manager.cc:1397] T 66bfa92db9874750b3c340e31a06c412 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.218s user 0.009s sys 0.007s
I20250114 02:54:20.066699 1206 raft_consensus.cc:357] T 66bfa92db9874750b3c340e31a06c412 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:20.067344 1206 raft_consensus.cc:383] T 66bfa92db9874750b3c340e31a06c412 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:20.067579 1206 raft_consensus.cc:738] T 66bfa92db9874750b3c340e31a06c412 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:20.072613 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet a3886c5f976445cebc95ef1839a74aa9 (table default.Table_35a78676616e4b20906b262c9d0c0d9e [id=e984149e9da444f1b2802e3f35a60fd6]) successfully deleted
I20250114 02:54:20.050333 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53a8bb487ad64157ad3baecf5d866eca" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
W20250114 02:54:20.077332 751 leader_election.cc:343] T 53a8bb487ad64157ad3baecf5d866eca P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:20.077726 751 leader_election.cc:304] T 53a8bb487ad64157ad3baecf5d866eca P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
W20250114 02:54:20.078260 751 raft_consensus.cc:2681] T 53a8bb487ad64157ad3baecf5d866eca P 819f9e3412e54d44abb11d8b3ca877c2: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250114 02:54:20.078281 933 tablet_service.cc:1467] Processing CreateTablet for tablet 59a46bc865cc48df8e98598764fd1b54 (DEFAULT_TABLE table=default.Table_1097f09fb10e441fb639f8d7aa12d487 [id=d8c9691818d042d7b49475fa009c4ca1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:20.079517 933 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59a46bc865cc48df8e98598764fd1b54. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:20.081440 934 tablet_service.cc:1467] Processing CreateTablet for tablet df045d80d8ce4ff194a6d2039a44bce6 (DEFAULT_TABLE table=default.Table_df476a4b13be4c11a896663f6211624d [id=da0258578bb34ae29551fc593bf7d0b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:20.084075 934 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df045d80d8ce4ff194a6d2039a44bce6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:20.088239 1206 consensus_queue.cc:260] T 66bfa92db9874750b3c340e31a06c412 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:20.090425 1206 ts_tablet_manager.cc:1428] T 66bfa92db9874750b3c340e31a06c412 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.026s user 0.004s sys 0.000s
I20250114 02:54:20.099464 1206 tablet_bootstrap.cc:492] T 0f1ea1608213492c9a216757896ff2a4 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:20.111783 936 tablet_service.cc:1467] Processing CreateTablet for tablet fc406eab968745d893f1a8538ff61f37 (DEFAULT_TABLE table=default.Table_debb45fd7c4b4826a73d39a6a6df88c4 [id=e1ca4114e55d48839dfa2f76624a4234]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:20.113333 936 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc406eab968745d893f1a8538ff61f37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:20.124521 1206 tablet_bootstrap.cc:654] T 0f1ea1608213492c9a216757896ff2a4 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:20.135352 1594 ts_tablet_manager.cc:1918] T 53a8bb487ad64157ad3baecf5d866eca P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:20.135727 1594 log.cc:1198] T 53a8bb487ad64157ad3baecf5d866eca P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/53a8bb487ad64157ad3baecf5d866eca
I20250114 02:54:20.136345 1594 ts_tablet_manager.cc:1939] T 53a8bb487ad64157ad3baecf5d866eca P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
W20250114 02:54:20.141171 1040 consensus_peers.cc:487] T 00080bea6c8e46e584ccba6dcb0e221b P 90d6c2c58bb54e0eafb365625ac33871 -> Peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Couldn't send request to peer 525198593c8e47afaee54e74267b87e3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 00080bea6c8e46e584ccba6dcb0e221b. This is attempt 1: this message will repeat every 5th retry.
I20250114 02:54:20.153807 1206 tablet_bootstrap.cc:492] T 0f1ea1608213492c9a216757896ff2a4 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:20.154213 1206 ts_tablet_manager.cc:1397] T 0f1ea1608213492c9a216757896ff2a4 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.055s user 0.008s sys 0.000s
I20250114 02:54:20.156531 1206 raft_consensus.cc:357] T 0f1ea1608213492c9a216757896ff2a4 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:20.157181 1206 raft_consensus.cc:383] T 0f1ea1608213492c9a216757896ff2a4 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:20.157436 1206 raft_consensus.cc:738] T 0f1ea1608213492c9a216757896ff2a4 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:20.157979 1206 consensus_queue.cc:260] T 0f1ea1608213492c9a216757896ff2a4 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:20.160022 1206 ts_tablet_manager.cc:1428] T 0f1ea1608213492c9a216757896ff2a4 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250114 02:54:20.161209 1595 ts_tablet_manager.cc:1918] T 961b6a25a13a4fa2a732a6383dcd8c84 P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:20.161654 1595 log.cc:1198] T 961b6a25a13a4fa2a732a6383dcd8c84 P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/961b6a25a13a4fa2a732a6383dcd8c84
I20250114 02:54:20.162698 1595 ts_tablet_manager.cc:1939] T 961b6a25a13a4fa2a732a6383dcd8c84 P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
W20250114 02:54:20.165287 1040 consensus_peers.cc:487] T 7ecf782bb3d54528aa996abca1ea3a15 P 90d6c2c58bb54e0eafb365625ac33871 -> Peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Couldn't send request to peer 525198593c8e47afaee54e74267b87e3. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250114 02:54:20.171557 1594 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 126) took 1721 ms. Trace:
I20250114 02:54:20.175377 1594 rpcz_store.cc:276] 0114 02:54:18.450495 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:18.450608 (+ 113us) service_pool.cc:225] Handling call
0114 02:54:20.056368 (+1605760us) ts_tablet_manager.cc:1145] Deleting tablet 53a8bb487ad64157ad3baecf5d866eca
0114 02:54:20.056404 (+ 36us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:20.128756 (+ 72352us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:20.135132 (+ 6376us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:20.171495 (+ 36363us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1605134}
I20250114 02:54:20.175835 1595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 128) took 3024 ms. Trace:
I20250114 02:54:20.176468 1594 tablet_replica.cc:331] T bc46815b4803418c95ea9c039a4fae88 P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:20.176231 1595 rpcz_store.cc:276] 0114 02:54:17.151091 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:17.151204 (+ 113us) service_pool.cc:225] Handling call
0114 02:54:19.843202 (+2691998us) ts_tablet_manager.cc:1145] Deleting tablet 961b6a25a13a4fa2a732a6383dcd8c84
0114 02:54:19.843246 (+ 44us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:20.034820 (+191574us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:20.160649 (+125829us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:20.175755 (+ 15106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2691496}
I20250114 02:54:20.177148 1594 raft_consensus.cc:2238] T bc46815b4803418c95ea9c039a4fae88 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:20.177485 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet 961b6a25a13a4fa2a732a6383dcd8c84 (table default.Table_0943b1ca347042c9a6043006bc566701 [id=e3418787ef0a4e22979fb41d4f94ebb0]) successfully deleted
I20250114 02:54:20.177584 1594 raft_consensus.cc:2267] T bc46815b4803418c95ea9c039a4fae88 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:20.178228 1595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 129) took 2911 ms. Trace:
I20250114 02:54:20.178642 1595 rpcz_store.cc:276] 0114 02:54:17.266743 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:17.266891 (+ 148us) service_pool.cc:225] Handling call
0114 02:54:20.177770 (+2910879us) ts_tablet_manager.cc:1145] Deleting tablet 66bfa92db9874750b3c340e31a06c412
0114 02:54:20.177826 (+ 56us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:20.178164 (+ 338us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2910215}
I20250114 02:54:20.179404 1594 ts_tablet_manager.cc:1905] T bc46815b4803418c95ea9c039a4fae88 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:20.183358 1206 tablet_bootstrap.cc:492] T 1381e26725174b9c877075cf5444f566 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
W20250114 02:54:20.200865 458 catalog_manager.cc:4852] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 66bfa92db9874750b3c340e31a06c412 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 66bfa92db9874750b3c340e31a06c412 already in progress: creating tablet
I20250114 02:54:20.202508 1595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 131) took 2898 ms. Trace:
I20250114 02:54:20.202942 1595 rpcz_store.cc:276] 0114 02:54:17.303913 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:17.304025 (+ 112us) service_pool.cc:225] Handling call
0114 02:54:20.202135 (+2898110us) ts_tablet_manager.cc:1145] Deleting tablet b4740f6b3a6741368ba68907ad065279
0114 02:54:20.202199 (+ 64us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:20.202436 (+ 237us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2897495}
I20250114 02:54:20.244575 1206 tablet_bootstrap.cc:654] T 1381e26725174b9c877075cf5444f566 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:20.245644 1652 raft_consensus.cc:491] T 55b68586ef8b45439a7dd8dcd2efcd79 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:20.245719 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet 53a8bb487ad64157ad3baecf5d866eca (table default.Table_5d3b333f1d99463b8ad3b4318c8f29df [id=0f1c410e9c5c40ab8b8fe4e4bf68e8ea]) successfully deleted
I20250114 02:54:20.246093 1652 raft_consensus.cc:513] T 55b68586ef8b45439a7dd8dcd2efcd79 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:20.247973 1652 leader_election.cc:290] T 55b68586ef8b45439a7dd8dcd2efcd79 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:20.248948 1652 raft_consensus.cc:491] T 266da813b4514329be0ad97d07b7114f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:20.249338 1652 raft_consensus.cc:513] T 266da813b4514329be0ad97d07b7114f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:20.251236 1652 leader_election.cc:290] T 266da813b4514329be0ad97d07b7114f P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:20.252393 959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "266da813b4514329be0ad97d07b7114f" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:20.253113 1099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55b68586ef8b45439a7dd8dcd2efcd79" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
W20250114 02:54:20.254458 751 leader_election.cc:343] T 266da813b4514329be0ad97d07b7114f P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:20.260186 1595 tablet_replica.cc:331] T 290ad07ab8be4791a39194909e000843 P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:20.261030 1595 raft_consensus.cc:2238] T 290ad07ab8be4791a39194909e000843 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:20.260983 1099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "266da813b4514329be0ad97d07b7114f" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:20.261652 1595 raft_consensus.cc:2267] T 290ad07ab8be4791a39194909e000843 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:20.263358 1595 ts_tablet_manager.cc:1905] T 290ad07ab8be4791a39194909e000843 P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250114 02:54:20.263334 458 catalog_manager.cc:4839] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet b4740f6b3a6741368ba68907ad065279 because the tablet was not found. No further retry: Not found: Tablet not found: b4740f6b3a6741368ba68907ad065279
W20250114 02:54:20.264523 753 leader_election.cc:343] T 55b68586ef8b45439a7dd8dcd2efcd79 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
W20250114 02:54:20.266358 753 leader_election.cc:343] T 266da813b4514329be0ad97d07b7114f P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:20.266738 753 leader_election.cc:304] T 266da813b4514329be0ad97d07b7114f P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:20.267338 1652 raft_consensus.cc:2743] T 266da813b4514329be0ad97d07b7114f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:20.277215 1206 tablet_bootstrap.cc:492] T 1381e26725174b9c877075cf5444f566 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:20.277761 1206 ts_tablet_manager.cc:1397] T 1381e26725174b9c877075cf5444f566 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.095s user 0.008s sys 0.003s
I20250114 02:54:20.284837 1206 raft_consensus.cc:357] T 1381e26725174b9c877075cf5444f566 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:20.285732 1206 raft_consensus.cc:383] T 1381e26725174b9c877075cf5444f566 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:20.286020 1206 raft_consensus.cc:738] T 1381e26725174b9c877075cf5444f566 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:20.288935 1594 ts_tablet_manager.cc:1918] T bc46815b4803418c95ea9c039a4fae88 P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:20.289362 1594 log.cc:1198] T bc46815b4803418c95ea9c039a4fae88 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/bc46815b4803418c95ea9c039a4fae88
I20250114 02:54:20.290063 1594 ts_tablet_manager.cc:1939] T bc46815b4803418c95ea9c039a4fae88 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:20.294219 939 tablet_service.cc:1467] Processing CreateTablet for tablet ed7f575653c848d7b076c17c38c4a244 (DEFAULT_TABLE table=default.Table_44d578b4433242d5b7d8a61f6e4213e7 [id=954669573a3c4a97a488805e05b0b220]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:20.293823 934 tablet_service.cc:1467] Processing CreateTablet for tablet 3161f75bc4e84000af022faa9804c810 (DEFAULT_TABLE table=default.Table_35a78676616e4b20906b262c9d0c0d9e [id=e984149e9da444f1b2802e3f35a60fd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:20.295166 939 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed7f575653c848d7b076c17c38c4a244. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:20.295660 934 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3161f75bc4e84000af022faa9804c810. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:20.296625 941 tablet_service.cc:1467] Processing CreateTablet for tablet 97f4e061d3b94ee9982410b7ce135453 (DEFAULT_TABLE table=default.Table_6a4fe39298514187b65def5841d3a992 [id=5c5a7974ad2f4e58ba5c47a6eeaaac62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:20.297468 941 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97f4e061d3b94ee9982410b7ce135453. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:20.298560 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55b68586ef8b45439a7dd8dcd2efcd79" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
W20250114 02:54:20.299719 751 leader_election.cc:343] T 55b68586ef8b45439a7dd8dcd2efcd79 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:20.300130 751 leader_election.cc:304] T 55b68586ef8b45439a7dd8dcd2efcd79 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:20.300799 1652 raft_consensus.cc:2743] T 55b68586ef8b45439a7dd8dcd2efcd79 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250114 02:54:20.302268 942 consensus_meta.cc:287] T 6e858a8449444ef999b9bbef612249d3 P 525198593c8e47afaee54e74267b87e3: Time spent flushing consensus metadata: real 0.526s user 0.004s sys 0.004s
I20250114 02:54:20.286713 1206 consensus_queue.cc:260] T 1381e26725174b9c877075cf5444f566 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:20.309675 1206 ts_tablet_manager.cc:1428] T 1381e26725174b9c877075cf5444f566 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.032s user 0.006s sys 0.002s
I20250114 02:54:20.310735 1206 tablet_bootstrap.cc:492] T 6448eea21ab2422c97da2494658ded3f P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:20.311233 1652 raft_consensus.cc:491] T 1381e26725174b9c877075cf5444f566 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:20.311703 1652 raft_consensus.cc:513] T 1381e26725174b9c877075cf5444f566 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:20.314605 1652 leader_election.cc:290] T 1381e26725174b9c877075cf5444f566 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:20.315372 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1381e26725174b9c877075cf5444f566" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
W20250114 02:54:20.316466 751 leader_election.cc:343] T 1381e26725174b9c877075cf5444f566 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:20.318950 1594 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 133) took 1633 ms. Trace:
I20250114 02:54:20.319370 1594 rpcz_store.cc:276] 0114 02:54:18.686025 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:18.686150 (+ 125us) service_pool.cc:225] Handling call
0114 02:54:20.176315 (+1490165us) ts_tablet_manager.cc:1145] Deleting tablet bc46815b4803418c95ea9c039a4fae88
0114 02:54:20.176365 (+ 50us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:20.207047 (+ 30682us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:20.288681 (+ 81634us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:20.318872 (+ 30191us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1489648}
I20250114 02:54:20.320938 1594 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 137) took 1235 ms. Trace:
I20250114 02:54:20.321292 1594 rpcz_store.cc:276] 0114 02:54:19.085768 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:19.085955 (+ 187us) service_pool.cc:225] Handling call
0114 02:54:20.320520 (+1234565us) ts_tablet_manager.cc:1145] Deleting tablet f0bd9bd5a17d418d8082ed0ebd1bf4cc
0114 02:54:20.320571 (+ 51us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:20.320883 (+ 312us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1233787}
I20250114 02:54:20.322412 1594 tablet_replica.cc:331] T 00080bea6c8e46e584ccba6dcb0e221b P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:20.323361 1594 raft_consensus.cc:2238] T 00080bea6c8e46e584ccba6dcb0e221b P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:20.323906 1594 raft_consensus.cc:2267] T 00080bea6c8e46e584ccba6dcb0e221b P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:20.326435 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet bc46815b4803418c95ea9c039a4fae88 (table default.Table_6a4fe39298514187b65def5841d3a992 [id=5c5a7974ad2f4e58ba5c47a6eeaaac62]) successfully deleted
W20250114 02:54:20.327064 458 catalog_manager.cc:4852] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet f0bd9bd5a17d418d8082ed0ebd1bf4cc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f0bd9bd5a17d418d8082ed0ebd1bf4cc already in progress: creating tablet
I20250114 02:54:20.315210 1101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1381e26725174b9c877075cf5444f566" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
W20250114 02:54:20.331235 753 leader_election.cc:343] T 1381e26725174b9c877075cf5444f566 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:20.331666 753 leader_election.cc:304] T 1381e26725174b9c877075cf5444f566 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:20.332648 1652 raft_consensus.cc:2743] T 1381e26725174b9c877075cf5444f566 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:20.330034 1207 tablet_bootstrap.cc:654] T 26328f8e32bc49ed9b224f0e08b9f949 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:20.343765 1205 tablet_bootstrap.cc:492] T 321b9ad6d1ac4d1c80c7a01faf2015af P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:20.344159 1205 ts_tablet_manager.cc:1397] T 321b9ad6d1ac4d1c80c7a01faf2015af P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.632s user 0.008s sys 0.004s
I20250114 02:54:20.346273 1205 raft_consensus.cc:357] T 321b9ad6d1ac4d1c80c7a01faf2015af P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:20.346879 1205 raft_consensus.cc:383] T 321b9ad6d1ac4d1c80c7a01faf2015af P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:20.347129 1205 raft_consensus.cc:738] T 321b9ad6d1ac4d1c80c7a01faf2015af P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:20.360977 465 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet 46360f726fad4534a2daf67581346fc1
I20250114 02:54:20.362501 465 catalog_manager.cc:6596] ReplaceTablet: tablet 46360f726fad4534a2daf67581346fc1 deleted and replaced by tablet be7aad41ccdd4ef9bcfe8a985155708f
I20250114 02:54:20.362888 465 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:33540 (request call id 192) took 1620 ms. Trace:
I20250114 02:54:20.369812 937 tablet_service.cc:1514] Processing DeleteTablet for tablet 46360f726fad4534a2daf67581346fc1 with delete_type TABLET_DATA_DELETED (replaced by tablet be7aad41ccdd4ef9bcfe8a985155708f) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:20.363224 465 rpcz_store.cc:276] 0114 02:54:18.742211 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:19.090455 (+348244us) service_pool.cc:225] Handling call
0114 02:54:20.362826 (+1272371us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:19.986902 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:19.987245 (+ 343us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:19.987268 (+ 23us) write_op.cc:435] Acquired schema lock
0114 02:54:19.987281 (+ 13us) tablet.cc:659] Decoding operations
0114 02:54:19.987598 (+ 317us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:19.987629 (+ 31us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:19.987649 (+ 20us) tablet.cc:682] Acquiring locks for 2 operations
0114 02:54:19.987841 (+ 192us) tablet.cc:698] Row locks acquired
0114 02:54:19.987854 (+ 13us) write_op.cc:260] PREPARE: finished
0114 02:54:20.017399 (+ 29545us) write_op.cc:270] Start()
0114 02:54:20.017495 (+ 96us) write_op.cc:276] Timestamp: P: 1736823260017236 usec, L: 0
0114 02:54:20.017512 (+ 17us) op_driver.cc:348] REPLICATION: starting
0114 02:54:20.017994 (+ 482us) log.cc:844] Serialized 625 byte log entry
0114 02:54:20.357026 (+339032us) op_driver.cc:464] REPLICATION: finished
0114 02:54:20.357667 (+ 641us) write_op.cc:301] APPLY: starting
0114 02:54:20.357736 (+ 69us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:20.357873 (+ 137us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:20.357988 (+ 115us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:20.358413 (+ 425us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:20.358678 (+ 265us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:20.358740 (+ 62us) write_op.cc:312] APPLY: finished
0114 02:54:20.358968 (+ 228us) log.cc:844] Serialized 29 byte log entry
0114 02:54:20.359146 (+ 178us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:20.359230 (+ 84us) write_op.cc:454] Released schema lock
0114 02:54:20.359271 (+ 41us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:20.359560 (+ 289us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:20.359594 (+ 34us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":456,"apply.run_cpu_time_us":1849,"apply.run_wall_time_us":1845,"num_ops":2,"prepare.queue_time_us":259733,"prepare.run_cpu_time_us":2311,"prepare.run_wall_time_us":223239,"raft.queue_time_us":110327,"raft.run_cpu_time_us":931,"raft.run_wall_time_us":942,"replication_time_us":339385}]]}
I20250114 02:54:20.387866 1207 tablet_bootstrap.cc:492] T 26328f8e32bc49ed9b224f0e08b9f949 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:20.388321 1207 ts_tablet_manager.cc:1397] T 26328f8e32bc49ed9b224f0e08b9f949 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.414s user 0.006s sys 0.004s
I20250114 02:54:20.347713 1205 consensus_queue.cc:260] T 321b9ad6d1ac4d1c80c7a01faf2015af P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:20.399835 1081 tablet_service.cc:1514] Processing DeleteTablet for tablet 46360f726fad4534a2daf67581346fc1 with delete_type TABLET_DATA_DELETED (replaced by tablet be7aad41ccdd4ef9bcfe8a985155708f) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:20.401165 789 tablet_service.cc:1514] Processing DeleteTablet for tablet 46360f726fad4534a2daf67581346fc1 with delete_type TABLET_DATA_DELETED (replaced by tablet be7aad41ccdd4ef9bcfe8a985155708f) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:20.407858 1205 ts_tablet_manager.cc:1428] T 321b9ad6d1ac4d1c80c7a01faf2015af P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.063s user 0.005s sys 0.000s
I20250114 02:54:20.408768 1205 tablet_bootstrap.cc:492] T 53a8bb487ad64157ad3baecf5d866eca P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:20.424183 1702 ts_tablet_manager.cc:1918] T 7ecf782bb3d54528aa996abca1ea3a15 P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:20.424593 1702 log.cc:1198] T 7ecf782bb3d54528aa996abca1ea3a15 P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/7ecf782bb3d54528aa996abca1ea3a15
I20250114 02:54:20.428506 1652 raft_consensus.cc:491] T 0f1ea1608213492c9a216757896ff2a4 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:20.429170 1652 raft_consensus.cc:513] T 0f1ea1608213492c9a216757896ff2a4 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:20.390748 1207 raft_consensus.cc:357] T 26328f8e32bc49ed9b224f0e08b9f949 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:20.433226 1207 raft_consensus.cc:383] T 26328f8e32bc49ed9b224f0e08b9f949 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:20.433482 1207 raft_consensus.cc:738] T 26328f8e32bc49ed9b224f0e08b9f949 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:20.437178 1207 consensus_queue.cc:260] T 26328f8e32bc49ed9b224f0e08b9f949 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:20.452962 1507 raft_consensus.cc:491] T ba215eb8422d4f879dd0bf5996543962 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:20.453410 1507 raft_consensus.cc:513] T ba215eb8422d4f879dd0bf5996543962 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:20.454272 1207 ts_tablet_manager.cc:1428] T 26328f8e32bc49ed9b224f0e08b9f949 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.066s user 0.004s sys 0.001s
I20250114 02:54:20.455051 1207 tablet_bootstrap.cc:492] T 66bfa92db9874750b3c340e31a06c412 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:20.455121 1507 leader_election.cc:290] T ba215eb8422d4f879dd0bf5996543962 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:20.463382 1702 ts_tablet_manager.cc:1939] T 7ecf782bb3d54528aa996abca1ea3a15 P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:20.464568 468 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet ca986f81ed6848a89588366f3f400eec
I20250114 02:54:20.464725 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba215eb8422d4f879dd0bf5996543962" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:20.465363 962 raft_consensus.cc:2388] T ba215eb8422d4f879dd0bf5996543962 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 90d6c2c58bb54e0eafb365625ac33871 in current term 1: Already voted for candidate 819f9e3412e54d44abb11d8b3ca877c2 in this term.
I20250114 02:54:20.465497 1702 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55068 (request call id 151) took 1156 ms. Trace:
I20250114 02:54:20.465987 468 catalog_manager.cc:6596] ReplaceTablet: tablet ca986f81ed6848a89588366f3f400eec deleted and replaced by tablet a2d68eccaaa64511a9ae36f2b07a7b59
I20250114 02:54:20.465751 1702 rpcz_store.cc:276] 0114 02:54:19.308947 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:19.314838 (+ 5891us) service_pool.cc:225] Handling call
0114 02:54:19.682514 (+367676us) ts_tablet_manager.cc:1145] Deleting tablet 7ecf782bb3d54528aa996abca1ea3a15
0114 02:54:19.682554 (+ 40us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:20.290771 (+608217us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:20.423945 (+133174us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:20.465428 (+ 41483us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":367053}
I20250114 02:54:20.466377 468 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:33540 (request call id 202) took 1032 ms. Trace:
I20250114 02:54:20.466972 1702 tablet_replica.cc:331] T 290ad07ab8be4791a39194909e000843 P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:20.469596 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet 7ecf782bb3d54528aa996abca1ea3a15 (table default.Table_b0cd93e5a498450e84fb0f1ba4be4505 [id=c70751b59bfc4c6fa60d0e8bb40d5f36]) successfully deleted
I20250114 02:54:20.470039 941 tablet_service.cc:1514] Processing DeleteTablet for tablet ca986f81ed6848a89588366f3f400eec with delete_type TABLET_DATA_DELETED (replaced by tablet a2d68eccaaa64511a9ae36f2b07a7b59) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:20.466711 468 rpcz_store.cc:276] 0114 02:54:19.433600 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:19.617748 (+184148us) service_pool.cc:225] Handling call
0114 02:54:19.664986 (+ 47238us) spinlock_profiling.cc:243] Waited 45.9 ms on lock 0x7b54000316d4. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616cad50 00007f646174df2a 00007f64633f2dec 00007f64634eed31 00007f6461270d47 00007f6461270cd2 00007f6461270c01 00007f6461270b7c 00007f646126fe32 00007f645f512dfc 00007f645f512236
0114 02:54:20.466305 (+801319us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:20.210391 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:20.210882 (+ 491us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:20.210915 (+ 33us) write_op.cc:435] Acquired schema lock
0114 02:54:20.210935 (+ 20us) tablet.cc:659] Decoding operations
0114 02:54:20.211347 (+ 412us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:20.211408 (+ 61us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:20.211442 (+ 34us) tablet.cc:682] Acquiring locks for 2 operations
0114 02:54:20.211735 (+ 293us) tablet.cc:698] Row locks acquired
0114 02:54:20.211754 (+ 19us) write_op.cc:260] PREPARE: finished
0114 02:54:20.211894 (+ 140us) write_op.cc:270] Start()
0114 02:54:20.212001 (+ 107us) write_op.cc:276] Timestamp: P: 1736823260211840 usec, L: 0
0114 02:54:20.212023 (+ 22us) op_driver.cc:348] REPLICATION: starting
0114 02:54:20.212686 (+ 663us) log.cc:844] Serialized 625 byte log entry
0114 02:54:20.457642 (+244956us) op_driver.cc:464] REPLICATION: finished
0114 02:54:20.458428 (+ 786us) write_op.cc:301] APPLY: starting
0114 02:54:20.458498 (+ 70us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:20.458625 (+ 127us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:20.458731 (+ 106us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:20.462215 (+ 3484us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:20.462581 (+ 366us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:20.462628 (+ 47us) write_op.cc:312] APPLY: finished
0114 02:54:20.462806 (+ 178us) log.cc:844] Serialized 29 byte log entry
0114 02:54:20.463517 (+ 711us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:20.463597 (+ 80us) write_op.cc:454] Released schema lock
0114 02:54:20.463638 (+ 41us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":105640704,"child_traces":[["op",{"apply.queue_time_us":597,"num_ops":2,"prepare.queue_time_us":481478,"prepare.run_cpu_time_us":2575,"prepare.run_wall_time_us":2575,"replication_time_us":245497}]]}
I20250114 02:54:20.479667 1207 tablet_bootstrap.cc:654] T 66bfa92db9874750b3c340e31a06c412 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:20.480598 1081 tablet_service.cc:1514] Processing DeleteTablet for tablet ca986f81ed6848a89588366f3f400eec with delete_type TABLET_DATA_DELETED (replaced by tablet a2d68eccaaa64511a9ae36f2b07a7b59) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:20.482369 1595 ts_tablet_manager.cc:1918] T 290ad07ab8be4791a39194909e000843 P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:20.448056 1205 tablet_bootstrap.cc:654] T 53a8bb487ad64157ad3baecf5d866eca P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:20.492295 1507 raft_consensus.cc:491] T 26328f8e32bc49ed9b224f0e08b9f949 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:20.492507 1595 log.cc:1198] T 290ad07ab8be4791a39194909e000843 P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/290ad07ab8be4791a39194909e000843
I20250114 02:54:20.492772 1507 raft_consensus.cc:513] T 26328f8e32bc49ed9b224f0e08b9f949 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:20.493914 1595 ts_tablet_manager.cc:1939] T 290ad07ab8be4791a39194909e000843 P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
I20250114 02:54:20.495186 1507 leader_election.cc:290] T 26328f8e32bc49ed9b224f0e08b9f949 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:20.495718 1702 raft_consensus.cc:2238] T 290ad07ab8be4791a39194909e000843 P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Raft consensus shutting down.
I20250114 02:54:20.496749 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26328f8e32bc49ed9b224f0e08b9f949" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:20.498265 471 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet de4ca83a69704a368a25954cdc1b9392 (table default.Table_3139140b7eec4849ae715c295fbeaa6b [id=8536eb7f6bf542daa88a8db0e4873456]). Expected version 2 got 0
W20250114 02:54:20.498693 1040 leader_election.cc:343] T 26328f8e32bc49ed9b224f0e08b9f949 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:20.499110 471 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet aad96f2507974023aaa988e6950feb7a (table default.Table_9676926a50eb42349a2d609b30566e9a [id=f38063ebf94d4084ab0174428503dc29]). Expected version 2 got 0
I20250114 02:54:20.507627 471 catalog_manager.cc:5526] T fcb58a05f980460884ed81a30a5d59c2 P 90d6c2c58bb54e0eafb365625ac33871 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67). New cstate: current_term: 1 leader_uuid: "90d6c2c58bb54e0eafb365625ac33871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } }
I20250114 02:54:20.497064 1702 raft_consensus.cc:2267] T 290ad07ab8be4791a39194909e000843 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:20.514734 1595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 132) took 3209 ms. Trace:
W20250114 02:54:20.505863 462 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.65:49805 (request call id 45) took 1180 ms (client timeout 1000 ms). Trace:
I20250114 02:54:20.525774 1205 tablet_bootstrap.cc:492] T 53a8bb487ad64157ad3baecf5d866eca P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:20.526158 1205 ts_tablet_manager.cc:1397] T 53a8bb487ad64157ad3baecf5d866eca P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.118s user 0.010s sys 0.000s
I20250114 02:54:20.528420 1595 rpcz_store.cc:276] 0114 02:54:17.305127 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:17.305264 (+ 137us) service_pool.cc:225] Handling call
0114 02:54:20.260019 (+2954755us) ts_tablet_manager.cc:1145] Deleting tablet 290ad07ab8be4791a39194909e000843
0114 02:54:20.260063 (+ 44us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:20.413150 (+153087us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:20.482092 (+ 68942us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:20.514655 (+ 32563us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2954200}
I20250114 02:54:20.530721 1702 ts_tablet_manager.cc:1905] T 290ad07ab8be4791a39194909e000843 P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:20.534022 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet 290ad07ab8be4791a39194909e000843 (table default.Table_02b3fc3bfe484e3a9c5d5e9114deb368 [id=29d8f8e2417d403aa619839418c94207]) successfully deleted
I20250114 02:54:20.534632 1595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 130) took 3231 ms. Trace:
I20250114 02:54:20.535017 1595 rpcz_store.cc:276] 0114 02:54:17.303391 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:17.308238 (+ 4847us) service_pool.cc:225] Handling call
0114 02:54:20.534307 (+3226069us) ts_tablet_manager.cc:1145] Deleting tablet 276942e5e54246c79f0d4f08297f58ea
0114 02:54:20.534360 (+ 53us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:20.534561 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3225429}
W20250114 02:54:20.536078 458 catalog_manager.cc:4839] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 276942e5e54246c79f0d4f08297f58ea because the tablet was not found. No further retry: Not found: Tablet not found: 276942e5e54246c79f0d4f08297f58ea
W20250114 02:54:20.515641 462 rpcz_store.cc:269] 0114 02:54:19.325417 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:19.343780 (+ 18363us) service_pool.cc:225] Handling call
0114 02:54:20.505778 (+1161998us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:20.213083 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:20.213421 (+ 338us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:20.213440 (+ 19us) write_op.cc:435] Acquired schema lock
0114 02:54:20.213452 (+ 12us) tablet.cc:659] Decoding operations
0114 02:54:20.213680 (+ 228us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:20.213711 (+ 31us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:20.213731 (+ 20us) tablet.cc:682] Acquiring locks for 1 operations
0114 02:54:20.213847 (+ 116us) tablet.cc:698] Row locks acquired
0114 02:54:20.213860 (+ 13us) write_op.cc:260] PREPARE: finished
0114 02:54:20.213952 (+ 92us) write_op.cc:270] Start()
0114 02:54:20.214019 (+ 67us) write_op.cc:276] Timestamp: P: 1736823260213913 usec, L: 0
0114 02:54:20.214032 (+ 13us) op_driver.cc:348] REPLICATION: starting
0114 02:54:20.214453 (+ 421us) log.cc:844] Serialized 530 byte log entry
0114 02:54:20.457914 (+243461us) op_driver.cc:464] REPLICATION: finished
0114 02:54:20.475926 (+ 18012us) write_op.cc:301] APPLY: starting
0114 02:54:20.475996 (+ 70us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:20.476106 (+ 110us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:20.476207 (+ 101us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:20.476415 (+ 208us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:20.476597 (+ 182us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:20.476660 (+ 63us) write_op.cc:312] APPLY: finished
0114 02:54:20.476868 (+ 208us) log.cc:844] Serialized 23 byte log entry
0114 02:54:20.482038 (+ 5170us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:20.482111 (+ 73us) write_op.cc:454] Released schema lock
0114 02:54:20.482148 (+ 37us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":5,"child_traces":[["op",{"apply.queue_time_us":17865,"num_ops":1,"prepare.queue_time_us":480327,"prepare.run_cpu_time_us":1545,"prepare.run_wall_time_us":1544,"replication_time_us":243800,"threads_started":1}]]}
I20250114 02:54:20.544533 471 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet a3fdab330d574c0283ac58bc9e450b87 (table default.Table_ec0eefb64458440ebd57a05a7ad0bb9a [id=b6874c22fe0d4151a7c8ea061f685938]). Expected version 1 got 0
I20250114 02:54:20.545367 471 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet f464eabfbc0945fa8a5583f00c1dab40 (table default.Table_4fd891b63924429085d78f78cb177f0e [id=d36c71e7562b4ff1bc8be5cf2955000e]). Expected version 1 got 0
I20250114 02:54:20.550539 1207 tablet_bootstrap.cc:492] T 66bfa92db9874750b3c340e31a06c412 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:20.551789 1207 ts_tablet_manager.cc:1397] T 66bfa92db9874750b3c340e31a06c412 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.097s user 0.007s sys 0.003s
I20250114 02:54:20.555032 1595 tablet_replica.cc:331] T a3886c5f976445cebc95ef1839a74aa9 P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:20.555732 1595 raft_consensus.cc:2238] T a3886c5f976445cebc95ef1839a74aa9 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:20.557593 1205 raft_consensus.cc:357] T 53a8bb487ad64157ad3baecf5d866eca P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:20.558283 1205 raft_consensus.cc:383] T 53a8bb487ad64157ad3baecf5d866eca P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:20.558606 1205 raft_consensus.cc:738] T 53a8bb487ad64157ad3baecf5d866eca P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:20.559340 1205 consensus_queue.cc:260] T 53a8bb487ad64157ad3baecf5d866eca P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
W20250114 02:54:20.562563 577 debug-util.cc:398] Leaking SignalData structure 0x7b08000abb20 after lost signal to thread 512
W20250114 02:54:20.563534 577 debug-util.cc:398] Leaking SignalData structure 0x7b08000b0b20 after lost signal to thread 580
I20250114 02:54:20.584036 1782 raft_consensus.cc:491] T 66bfa92db9874750b3c340e31a06c412 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:20.584548 1782 raft_consensus.cc:513] T 66bfa92db9874750b3c340e31a06c412 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:20.585095 1594 ts_tablet_manager.cc:1905] T 00080bea6c8e46e584ccba6dcb0e221b P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:20.586345 1782 leader_election.cc:290] T 66bfa92db9874750b3c340e31a06c412 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:20.588447 1101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66bfa92db9874750b3c340e31a06c412" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:20.589484 1652 leader_election.cc:290] T 0f1ea1608213492c9a216757896ff2a4 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:20.591368 1595 raft_consensus.cc:2267] T a3886c5f976445cebc95ef1839a74aa9 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:20.594714 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26328f8e32bc49ed9b224f0e08b9f949" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:20.595242 818 raft_consensus.cc:2463] T 26328f8e32bc49ed9b224f0e08b9f949 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 0.
I20250114 02:54:20.596057 959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66bfa92db9874750b3c340e31a06c412" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:20.596431 789 tablet_service.cc:1514] Processing DeleteTablet for tablet ca986f81ed6848a89588366f3f400eec with delete_type TABLET_DATA_DELETED (replaced by tablet a2d68eccaaa64511a9ae36f2b07a7b59) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:20.597092 961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f1ea1608213492c9a216757896ff2a4" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:20.598263 816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba215eb8422d4f879dd0bf5996543962" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:20.599949 1595 ts_tablet_manager.cc:1905] T a3886c5f976445cebc95ef1839a74aa9 P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:20.600579 1207 raft_consensus.cc:357] T 66bfa92db9874750b3c340e31a06c412 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:20.601209 1207 raft_consensus.cc:383] T 66bfa92db9874750b3c340e31a06c412 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:20.601471 1207 raft_consensus.cc:738] T 66bfa92db9874750b3c340e31a06c412 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:20.601567 1042 leader_election.cc:304] T 26328f8e32bc49ed9b224f0e08b9f949 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters: 525198593c8e47afaee54e74267b87e3
I20250114 02:54:20.602448 1507 raft_consensus.cc:2798] T 26328f8e32bc49ed9b224f0e08b9f949 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:20.602734 1507 raft_consensus.cc:491] T 26328f8e32bc49ed9b224f0e08b9f949 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:20.602996 1507 raft_consensus.cc:3054] T 26328f8e32bc49ed9b224f0e08b9f949 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:20.610603 1507 raft_consensus.cc:513] T 26328f8e32bc49ed9b224f0e08b9f949 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:20.612488 1207 consensus_queue.cc:260] T 66bfa92db9874750b3c340e31a06c412 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:20.613009 1042 leader_election.cc:304] T ba215eb8422d4f879dd0bf5996543962 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90d6c2c58bb54e0eafb365625ac33871; no voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2
I20250114 02:54:20.614068 1721 raft_consensus.cc:3054] T ba215eb8422d4f879dd0bf5996543962 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
W20250114 02:54:20.623868 751 leader_election.cc:343] T 66bfa92db9874750b3c340e31a06c412 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:20.624779 1721 raft_consensus.cc:2743] T ba215eb8422d4f879dd0bf5996543962 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250114 02:54:20.627518 751 leader_election.cc:343] T 0f1ea1608213492c9a216757896ff2a4 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:20.632565 1206 tablet_bootstrap.cc:654] T 6448eea21ab2422c97da2494658ded3f P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:20.633591 1741 consensus_queue.cc:1035] T ba215eb8422d4f879dd0bf5996543962 P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:20.635355 816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26328f8e32bc49ed9b224f0e08b9f949" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:20.636387 816 raft_consensus.cc:3054] T 26328f8e32bc49ed9b224f0e08b9f949 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
W20250114 02:54:20.652046 753 leader_election.cc:343] T 66bfa92db9874750b3c340e31a06c412 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:20.652436 753 leader_election.cc:304] T 66bfa92db9874750b3c340e31a06c412 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:20.652377 1101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f1ea1608213492c9a216757896ff2a4" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
W20250114 02:54:20.653558 753 leader_election.cc:343] T 0f1ea1608213492c9a216757896ff2a4 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:20.653874 753 leader_election.cc:304] T 0f1ea1608213492c9a216757896ff2a4 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:20.654729 1507 leader_election.cc:290] T 26328f8e32bc49ed9b224f0e08b9f949 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Requested vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:20.655974 1207 ts_tablet_manager.cc:1428] T 66bfa92db9874750b3c340e31a06c412 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.104s user 0.001s sys 0.005s
I20250114 02:54:20.656425 959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26328f8e32bc49ed9b224f0e08b9f949" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
W20250114 02:54:20.657761 1040 leader_election.cc:343] T 26328f8e32bc49ed9b224f0e08b9f949 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
W20250114 02:54:20.665146 455 consensus_peers.cc:487] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750 -> Peer 87b3db09ac87436e9a60d6de5fedaff5 (127.0.107.124:39105): Couldn't send request to peer 87b3db09ac87436e9a60d6de5fedaff5. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.124:39105: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20250114 02:54:20.651098 816 raft_consensus.cc:2463] T 26328f8e32bc49ed9b224f0e08b9f949 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1.
I20250114 02:54:20.672755 1207 tablet_bootstrap.cc:492] T 55b68586ef8b45439a7dd8dcd2efcd79 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
W20250114 02:54:20.693754 1042 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.65:39397, user_credentials={real_user=slave}} blocked reactor thread for 80813.5us
I20250114 02:54:20.704665 1042 leader_election.cc:304] T 26328f8e32bc49ed9b224f0e08b9f949 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters: 525198593c8e47afaee54e74267b87e3
I20250114 02:54:20.705649 1721 raft_consensus.cc:2798] T 26328f8e32bc49ed9b224f0e08b9f949 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:20.706117 1721 raft_consensus.cc:695] T 26328f8e32bc49ed9b224f0e08b9f949 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Becoming Leader. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Running, Role: LEADER
I20250114 02:54:20.706885 1721 consensus_queue.cc:237] T 26328f8e32bc49ed9b224f0e08b9f949 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:20.709378 466 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_d101e66d79a64b6f9233687a3cb522a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:20.711337 466 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d101e66d79a64b6f9233687a3cb522a3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:20.714148 1780 raft_consensus.cc:2743] T 66bfa92db9874750b3c340e31a06c412 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:20.714764 1730 raft_consensus.cc:2743] T 0f1ea1608213492c9a216757896ff2a4 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250114 02:54:20.715602 753 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.67:45749, user_credentials={real_user=slave}} blocked reactor thread for 62065.8us
W20250114 02:54:20.719007 464 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6e858a8449444ef999b9bbef612249d3 consensus state has no leader
I20250114 02:54:20.720458 464 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_e2ae3006a8ca4082aee1af910f435789"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:20.722050 1205 ts_tablet_manager.cc:1428] T 53a8bb487ad64157ad3baecf5d866eca P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.196s user 0.007s sys 0.000s
W20250114 02:54:20.722249 464 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e2ae3006a8ca4082aee1af910f435789 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:20.723466 1205 tablet_bootstrap.cc:492] T 1381e26725174b9c877075cf5444f566 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:20.728426 1784 raft_consensus.cc:491] T 66bfa92db9874750b3c340e31a06c412 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:20.729274 1784 raft_consensus.cc:513] T 66bfa92db9874750b3c340e31a06c412 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:20.733225 1784 leader_election.cc:290] T 66bfa92db9874750b3c340e31a06c412 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:20.733162 959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66bfa92db9874750b3c340e31a06c412" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
W20250114 02:54:20.734402 1040 leader_election.cc:343] T 66bfa92db9874750b3c340e31a06c412 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:20.748478 816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66bfa92db9874750b3c340e31a06c412" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:20.749207 816 raft_consensus.cc:2463] T 66bfa92db9874750b3c340e31a06c412 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 0.
I20250114 02:54:20.753679 1042 leader_election.cc:304] T 66bfa92db9874750b3c340e31a06c412 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters: 525198593c8e47afaee54e74267b87e3
I20250114 02:54:20.754845 1507 raft_consensus.cc:2798] T 66bfa92db9874750b3c340e31a06c412 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:20.755221 1507 raft_consensus.cc:491] T 66bfa92db9874750b3c340e31a06c412 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:20.755602 1507 raft_consensus.cc:3054] T 66bfa92db9874750b3c340e31a06c412 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
W20250114 02:54:20.765010 455 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 044832851f804208b0a371eb0857dbf3 (table default.Table_0518d8bed7304c86aff32e05e6f07c0d [id=515a2fe3f1e84e82bccc02ef5cecef1b], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 044832851f804208b0a371eb0857dbf3 consensus state has no leader
I20250114 02:54:20.766635 1207 tablet_bootstrap.cc:654] T 55b68586ef8b45439a7dd8dcd2efcd79 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:20.774163 1702 ts_tablet_manager.cc:1918] T 290ad07ab8be4791a39194909e000843 P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:20.774561 1702 log.cc:1198] T 290ad07ab8be4791a39194909e000843 P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/290ad07ab8be4791a39194909e000843
I20250114 02:54:20.775374 1702 ts_tablet_manager.cc:1939] T 290ad07ab8be4791a39194909e000843 P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:20.775702 1206 tablet_bootstrap.cc:492] T 6448eea21ab2422c97da2494658ded3f P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:20.776130 1206 ts_tablet_manager.cc:1397] T 6448eea21ab2422c97da2494658ded3f P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.466s user 0.005s sys 0.008s
I20250114 02:54:20.799674 1652 consensus_queue.cc:1035] T ba215eb8422d4f879dd0bf5996543962 P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:20.806632 1205 tablet_bootstrap.cc:654] T 1381e26725174b9c877075cf5444f566 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:20.811328 1595 ts_tablet_manager.cc:1918] T a3886c5f976445cebc95ef1839a74aa9 P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250114 02:54:20.811723 1595 log.cc:1198] T a3886c5f976445cebc95ef1839a74aa9 P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/a3886c5f976445cebc95ef1839a74aa9
I20250114 02:54:20.811705 463 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33540 (request call id 204) took 1025 ms. Trace:
W20250114 02:54:20.811964 470 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 92cd8eccacf34ab7a66b3bbc69df3697 consensus state has no leader
I20250114 02:54:20.812834 1595 ts_tablet_manager.cc:1939] T a3886c5f976445cebc95ef1839a74aa9 P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
I20250114 02:54:20.812000 463 rpcz_store.cc:276] 0114 02:54:19.786238 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:19.786677 (+ 439us) service_pool.cc:225] Handling call
0114 02:54:19.904913 (+118236us) spinlock_profiling.cc:243] Waited 117 ms on lock 0x7b54000316d4. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616cad50 00007f646174df2a 00007f64633f2dec 00007f64634ec306 00007f6461250044 00007f646124ffd2 00007f646124ff01 00007f646124fe7c 00007f646124f132 00007f645f512dfc 00007f645f512236
0114 02:54:19.907162 (+ 2249us) catalog_manager.cc:2076] Acquired catalog manager lock
0114 02:54:19.909123 (+ 1961us) catalog_manager.cc:2122] Created new table and tablet info
0114 02:54:20.811135 (+902012us) catalog_manager.cc:2182] Wrote table and tablets to system table
0114 02:54:20.811425 (+ 290us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0114 02:54:20.811654 (+ 229us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:20.602286 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:20.602975 (+ 689us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:20.603033 (+ 58us) write_op.cc:435] Acquired schema lock
0114 02:54:20.603052 (+ 19us) tablet.cc:659] Decoding operations
0114 02:54:20.638251 (+ 35199us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:20.638301 (+ 50us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:20.638334 (+ 33us) tablet.cc:682] Acquiring locks for 2 operations
0114 02:54:20.638595 (+ 261us) tablet.cc:698] Row locks acquired
0114 02:54:20.638613 (+ 18us) write_op.cc:260] PREPARE: finished
0114 02:54:20.638745 (+ 132us) write_op.cc:270] Start()
0114 02:54:20.638843 (+ 98us) write_op.cc:276] Timestamp: P: 1736823260638654 usec, L: 0
0114 02:54:20.638863 (+ 20us) op_driver.cc:348] REPLICATION: starting
0114 02:54:20.639451 (+ 588us) log.cc:844] Serialized 433 byte log entry
0114 02:54:20.707540 (+ 68089us) spinlock_profiling.cc:243] Waited 46 ms on lock 0x7b5400032120. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616cad50 00007f64616d5288 00007f64616d4eca 00007f6461740cd5 00007f6461741179 00007f6461bad3b1 00007f6461bacb88 00007f6461bb1821 00007f6461bb17d9 00007f6461bb1769 00007f6461bb1731
0114 02:54:20.807800 (+100260us) op_driver.cc:464] REPLICATION: finished
0114 02:54:20.809687 (+ 1887us) write_op.cc:301] APPLY: starting
0114 02:54:20.809740 (+ 53us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:20.809857 (+ 117us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:20.809884 (+ 27us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:20.810335 (+ 451us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:20.810500 (+ 165us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 02:54:20.810538 (+ 38us) write_op.cc:312] APPLY: finished
0114 02:54:20.810683 (+ 145us) log.cc:844] Serialized 29 byte log entry
0114 02:54:20.810822 (+ 139us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:20.810906 (+ 84us) write_op.cc:454] Released schema lock
0114 02:54:20.810941 (+ 35us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":269661056,"child_traces":[["op",{"apply.queue_time_us":1728,"apply.run_cpu_time_us":1459,"apply.run_wall_time_us":1459,"num_ops":2,"prepare.queue_time_us":691999,"prepare.run_cpu_time_us":3627,"prepare.run_wall_time_us":135475,"raft.queue_time_us":113,"raft.run_cpu_time_us":14,"raft.run_wall_time_us":15,"replication_time_us":168853,"spinlock_wait_cycles":105889152}]]}
I20250114 02:54:20.815868 463 master_service.cc:946] ReplaceTablet: received request to replace tablet a456420f78714cf895528e19dfe0ee8d from {username='slave'} at 127.0.0.1:33540
I20250114 02:54:20.816279 1702 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55068 (request call id 150) took 1507 ms. Trace:
I20250114 02:54:20.816618 1702 rpcz_store.cc:276] 0114 02:54:19.308701 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:19.308822 (+ 121us) service_pool.cc:225] Handling call
0114 02:54:20.466822 (+1158000us) ts_tablet_manager.cc:1145] Deleting tablet 290ad07ab8be4791a39194909e000843
0114 02:54:20.466866 (+ 44us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:20.641971 (+175105us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:20.773925 (+131954us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:20.816196 (+ 42271us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1149698}
I20250114 02:54:20.817852 1702 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55068 (request call id 152) took 1508 ms. Trace:
I20250114 02:54:20.818125 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet 290ad07ab8be4791a39194909e000843 (table default.Table_02b3fc3bfe484e3a9c5d5e9114deb368 [id=29d8f8e2417d403aa619839418c94207]) successfully deleted
I20250114 02:54:20.818140 1702 rpcz_store.cc:276] 0114 02:54:19.309186 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:19.309316 (+ 130us) service_pool.cc:225] Handling call
0114 02:54:20.817559 (+1508243us) ts_tablet_manager.cc:1145] Deleting tablet 9589afd0094545c1801b7a0b5eb8698f
0114 02:54:20.817598 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:20.817796 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1500271}
I20250114 02:54:20.819110 1702 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55068 (request call id 153) took 1509 ms. Trace:
W20250114 02:54:20.819312 456 catalog_manager.cc:4839] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet 9589afd0094545c1801b7a0b5eb8698f because the tablet was not found. No further retry: Not found: Tablet not found: 9589afd0094545c1801b7a0b5eb8698f
I20250114 02:54:20.819373 1702 rpcz_store.cc:276] 0114 02:54:19.309402 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:19.309836 (+ 434us) service_pool.cc:225] Handling call
0114 02:54:20.818878 (+1509042us) ts_tablet_manager.cc:1145] Deleting tablet 276942e5e54246c79f0d4f08297f58ea
0114 02:54:20.818917 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:20.819067 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1501497}
W20250114 02:54:20.821396 456 catalog_manager.cc:4839] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet 276942e5e54246c79f0d4f08297f58ea because the tablet was not found. No further retry: Not found: Tablet not found: 276942e5e54246c79f0d4f08297f58ea
I20250114 02:54:20.823164 470 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_4598efdd6ff348d081ed8c3e7dc3130f" } modify_external_catalogs: true
I20250114 02:54:20.823627 470 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_4598efdd6ff348d081ed8c3e7dc3130f" } modify_external_catalogs: true
I20250114 02:54:20.794276 1507 raft_consensus.cc:513] T 66bfa92db9874750b3c340e31a06c412 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:20.826814 1786 raft_consensus.cc:491] T 53a8bb487ad64157ad3baecf5d866eca P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:20.827809 1786 raft_consensus.cc:513] T 53a8bb487ad64157ad3baecf5d866eca P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:20.829011 816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66bfa92db9874750b3c340e31a06c412" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:20.829545 816 raft_consensus.cc:3054] T 66bfa92db9874750b3c340e31a06c412 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:20.830286 1786 leader_election.cc:290] T 53a8bb487ad64157ad3baecf5d866eca P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
W20250114 02:54:20.831758 898 leader_election.cc:343] T 53a8bb487ad64157ad3baecf5d866eca P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Not found: Tablet not found: 53a8bb487ad64157ad3baecf5d866eca
I20250114 02:54:20.833818 1595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 134) took 2466 ms. Trace:
I20250114 02:54:20.833976 1207 tablet_bootstrap.cc:492] T 55b68586ef8b45439a7dd8dcd2efcd79 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:20.834388 1207 ts_tablet_manager.cc:1397] T 55b68586ef8b45439a7dd8dcd2efcd79 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.162s user 0.008s sys 0.000s
I20250114 02:54:20.836619 1207 raft_consensus.cc:357] T 55b68586ef8b45439a7dd8dcd2efcd79 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:20.836795 816 raft_consensus.cc:2463] T 66bfa92db9874750b3c340e31a06c412 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1.
I20250114 02:54:20.834129 1595 rpcz_store.cc:276] 0114 02:54:18.367120 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:18.367240 (+ 120us) service_pool.cc:225] Handling call
0114 02:54:20.554861 (+2187621us) ts_tablet_manager.cc:1145] Deleting tablet a3886c5f976445cebc95ef1839a74aa9
0114 02:54:20.554913 (+ 52us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:20.692359 (+137446us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:20.811095 (+118736us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:20.833764 (+ 22669us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2187066}
I20250114 02:54:20.838366 1042 leader_election.cc:304] T 66bfa92db9874750b3c340e31a06c412 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:20.838739 1207 raft_consensus.cc:383] T 55b68586ef8b45439a7dd8dcd2efcd79 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:20.838958 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet a3886c5f976445cebc95ef1839a74aa9 (table default.Table_35a78676616e4b20906b262c9d0c0d9e [id=e984149e9da444f1b2802e3f35a60fd6]) successfully deleted
I20250114 02:54:20.839042 1207 raft_consensus.cc:738] T 55b68586ef8b45439a7dd8dcd2efcd79 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:20.839557 1207 consensus_queue.cc:260] T 55b68586ef8b45439a7dd8dcd2efcd79 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:20.839902 1099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53a8bb487ad64157ad3baecf5d866eca" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:20.840435 1784 raft_consensus.cc:2798] T 66bfa92db9874750b3c340e31a06c412 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election won for term 1
W20250114 02:54:20.841034 898 leader_election.cc:343] T 53a8bb487ad64157ad3baecf5d866eca P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:20.841531 1207 ts_tablet_manager.cc:1428] T 55b68586ef8b45439a7dd8dcd2efcd79 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250114 02:54:20.841463 898 leader_election.cc:304] T 53a8bb487ad64157ad3baecf5d866eca P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 525198593c8e47afaee54e74267b87e3; no voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:20.842257 1786 raft_consensus.cc:2743] T 53a8bb487ad64157ad3baecf5d866eca P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:20.839862 1595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 135) took 2392 ms. Trace:
I20250114 02:54:20.844610 1595 rpcz_store.cc:276] 0114 02:54:18.447888 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:18.448004 (+ 116us) service_pool.cc:225] Handling call
0114 02:54:20.839559 (+2391555us) ts_tablet_manager.cc:1145] Deleting tablet 53a8bb487ad64157ad3baecf5d866eca
0114 02:54:20.839592 (+ 33us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:20.839816 (+ 224us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2390649}
I20250114 02:54:20.853785 467 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet a3fdab330d574c0283ac58bc9e450b87 (table default.Table_ec0eefb64458440ebd57a05a7ad0bb9a [id=b6874c22fe0d4151a7c8ea061f685938]). Expected version 1 got 0
I20250114 02:54:20.854753 467 catalog_manager.cc:5526] T f464eabfbc0945fa8a5583f00c1dab40 P 525198593c8e47afaee54e74267b87e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 525198593c8e47afaee54e74267b87e3 (127.0.107.66). New cstate: current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:20.856715 961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66bfa92db9874750b3c340e31a06c412" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
I20250114 02:54:20.846252 1507 leader_election.cc:290] T 66bfa92db9874750b3c340e31a06c412 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Requested vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:20.863969 1702 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55068 (request call id 146) took 1672 ms. Trace:
W20250114 02:54:20.858004 1040 leader_election.cc:343] T 66bfa92db9874750b3c340e31a06c412 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:20.851656 1207 tablet_bootstrap.cc:492] T 321b9ad6d1ac4d1c80c7a01faf2015af P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
W20250114 02:54:20.865973 458 catalog_manager.cc:4852] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 53a8bb487ad64157ad3baecf5d866eca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 53a8bb487ad64157ad3baecf5d866eca already in progress: creating tablet
I20250114 02:54:20.868424 464 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_579071b88792449ab70a258304f3356c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:20.870066 464 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_579071b88792449ab70a258304f3356c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:20.871019 1207 tablet_bootstrap.cc:654] T 321b9ad6d1ac4d1c80c7a01faf2015af P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:20.872439 467 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet f464eabfbc0945fa8a5583f00c1dab40 (table default.Table_4fd891b63924429085d78f78cb177f0e [id=d36c71e7562b4ff1bc8be5cf2955000e]). Expected version 1 got 0
I20250114 02:54:20.864178 1595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 143) took 2179 ms. Trace:
I20250114 02:54:20.872969 1595 rpcz_store.cc:276] 0114 02:54:18.685103 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:18.693324 (+ 8221us) service_pool.cc:225] Handling call
0114 02:54:20.863858 (+2170534us) ts_tablet_manager.cc:1145] Deleting tablet bc46815b4803418c95ea9c039a4fae88
0114 02:54:20.863897 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:20.864132 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2170020}
I20250114 02:54:20.864346 1702 rpcz_store.cc:276] 0114 02:54:19.191145 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:19.306374 (+115229us) service_pool.cc:225] Handling call
0114 02:54:20.820191 (+1513817us) ts_tablet_manager.cc:1145] Deleting tablet f0bd9bd5a17d418d8082ed0ebd1bf4cc
0114 02:54:20.863619 (+ 43428us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:20.863910 (+ 291us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1502576}
W20250114 02:54:20.874336 456 catalog_manager.cc:4852] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet f0bd9bd5a17d418d8082ed0ebd1bf4cc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f0bd9bd5a17d418d8082ed0ebd1bf4cc already in progress: creating tablet
W20250114 02:54:20.875375 458 catalog_manager.cc:4852] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet bc46815b4803418c95ea9c039a4fae88 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc46815b4803418c95ea9c039a4fae88 already in progress: creating tablet
I20250114 02:54:20.875840 1595 tablet_replica.cc:331] T 00080bea6c8e46e584ccba6dcb0e221b P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:20.877523 1595 raft_consensus.cc:2238] T 00080bea6c8e46e584ccba6dcb0e221b P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Raft consensus shutting down.
I20250114 02:54:20.880437 1595 raft_consensus.cc:2267] T 00080bea6c8e46e584ccba6dcb0e221b P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:20.882287 1205 tablet_bootstrap.cc:492] T 1381e26725174b9c877075cf5444f566 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:20.882823 1205 ts_tablet_manager.cc:1397] T 1381e26725174b9c877075cf5444f566 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.160s user 0.010s sys 0.002s
I20250114 02:54:20.884447 1595 ts_tablet_manager.cc:1905] T 00080bea6c8e46e584ccba6dcb0e221b P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:20.875303 1702 tablet_replica.cc:331] T ba215eb8422d4f879dd0bf5996543962 P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:20.884327 1784 raft_consensus.cc:695] T 66bfa92db9874750b3c340e31a06c412 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Becoming Leader. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Running, Role: LEADER
I20250114 02:54:20.888940 1784 consensus_queue.cc:237] T 66bfa92db9874750b3c340e31a06c412 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:20.891510 1206 raft_consensus.cc:357] T 6448eea21ab2422c97da2494658ded3f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:20.892174 1206 raft_consensus.cc:383] T 6448eea21ab2422c97da2494658ded3f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:20.892454 1206 raft_consensus.cc:738] T 6448eea21ab2422c97da2494658ded3f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:20.893069 1206 consensus_queue.cc:260] T 6448eea21ab2422c97da2494658ded3f P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:20.893875 1702 raft_consensus.cc:2238] T ba215eb8422d4f879dd0bf5996543962 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:20.894382 1702 raft_consensus.cc:2267] T ba215eb8422d4f879dd0bf5996543962 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:20.895016 1206 ts_tablet_manager.cc:1428] T 6448eea21ab2422c97da2494658ded3f P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.119s user 0.004s sys 0.000s
I20250114 02:54:20.900487 1702 ts_tablet_manager.cc:1905] T ba215eb8422d4f879dd0bf5996543962 P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:20.911365 1205 raft_consensus.cc:357] T 1381e26725174b9c877075cf5444f566 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:20.912096 1205 raft_consensus.cc:383] T 1381e26725174b9c877075cf5444f566 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:20.912415 1205 raft_consensus.cc:738] T 1381e26725174b9c877075cf5444f566 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:20.926677 1595 ts_tablet_manager.cc:1918] T 00080bea6c8e46e584ccba6dcb0e221b P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:20.933244 1205 consensus_queue.cc:260] T 1381e26725174b9c877075cf5444f566 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:20.939740 1205 ts_tablet_manager.cc:1428] T 1381e26725174b9c877075cf5444f566 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.057s user 0.007s sys 0.000s
I20250114 02:54:20.941450 1595 log.cc:1198] T 00080bea6c8e46e584ccba6dcb0e221b P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/00080bea6c8e46e584ccba6dcb0e221b
I20250114 02:54:20.943305 1595 ts_tablet_manager.cc:1939] T 00080bea6c8e46e584ccba6dcb0e221b P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
I20250114 02:54:20.943750 1205 tablet_bootstrap.cc:492] T 66bfa92db9874750b3c340e31a06c412 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:20.947475 1595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 147) took 1843 ms. Trace:
I20250114 02:54:20.951449 1205 tablet_bootstrap.cc:654] T 66bfa92db9874750b3c340e31a06c412 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:20.953831 1595 rpcz_store.cc:276] 0114 02:54:19.103721 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:19.103860 (+ 139us) service_pool.cc:225] Handling call
0114 02:54:20.875699 (+1771839us) ts_tablet_manager.cc:1145] Deleting tablet 00080bea6c8e46e584ccba6dcb0e221b
0114 02:54:20.875735 (+ 36us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:20.894633 (+ 18898us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:20.926479 (+ 31846us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:20.947419 (+ 20940us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1771205}
I20250114 02:54:20.957199 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet 00080bea6c8e46e584ccba6dcb0e221b (table default.Table_eb7878bdce82445c97f6ce665eb52d8e [id=3adecb85db3b44a4a65e50e61a91e908]) successfully deleted
I20250114 02:54:20.958596 1595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 148) took 1854 ms. Trace:
I20250114 02:54:20.961964 1595 rpcz_store.cc:276] 0114 02:54:19.104033 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:19.104247 (+ 214us) service_pool.cc:225] Handling call
0114 02:54:20.958272 (+1854025us) ts_tablet_manager.cc:1145] Deleting tablet f0bd9bd5a17d418d8082ed0ebd1bf4cc
0114 02:54:20.958318 (+ 46us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:20.958547 (+ 229us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1853389}
W20250114 02:54:20.965816 458 catalog_manager.cc:4852] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet f0bd9bd5a17d418d8082ed0ebd1bf4cc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f0bd9bd5a17d418d8082ed0ebd1bf4cc already in progress: creating tablet
I20250114 02:54:20.967108 1595 tablet_replica.cc:331] T ba215eb8422d4f879dd0bf5996543962 P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:20.969467 1595 raft_consensus.cc:2238] T ba215eb8422d4f879dd0bf5996543962 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:20.972535 1595 raft_consensus.cc:2267] T ba215eb8422d4f879dd0bf5996543962 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:20.978551 1595 ts_tablet_manager.cc:1905] T ba215eb8422d4f879dd0bf5996543962 P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:20.984088 1702 ts_tablet_manager.cc:1918] T ba215eb8422d4f879dd0bf5996543962 P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:20.984771 1702 log.cc:1198] T ba215eb8422d4f879dd0bf5996543962 P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/ba215eb8422d4f879dd0bf5996543962
I20250114 02:54:20.994318 1206 tablet_bootstrap.cc:492] T 044832851f804208b0a371eb0857dbf3 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:20.994491 1702 ts_tablet_manager.cc:1939] T ba215eb8422d4f879dd0bf5996543962 P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:20.996431 1207 tablet_bootstrap.cc:492] T 321b9ad6d1ac4d1c80c7a01faf2015af P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:20.997274 1702 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55068 (request call id 149) took 1804 ms. Trace:
I20250114 02:54:20.997660 1702 rpcz_store.cc:276] 0114 02:54:19.308408 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:19.319055 (+ 10647us) service_pool.cc:225] Handling call
0114 02:54:20.875098 (+1556043us) ts_tablet_manager.cc:1145] Deleting tablet ba215eb8422d4f879dd0bf5996543962
0114 02:54:20.875156 (+ 58us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:20.966218 (+ 91062us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:20.983871 (+ 17653us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:20.997207 (+ 13336us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1539253}
I20250114 02:54:20.998903 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet ba215eb8422d4f879dd0bf5996543962 (table default.Table_bacdf22532c84100a69a38cfa0ad07e5 [id=86304f6b48fd421cb953070ae7c9b0a2]) successfully deleted
I20250114 02:54:21.000114 1207 ts_tablet_manager.cc:1397] T 321b9ad6d1ac4d1c80c7a01faf2015af P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.149s user 0.016s sys 0.018s
I20250114 02:54:21.002542 1207 raft_consensus.cc:357] T 321b9ad6d1ac4d1c80c7a01faf2015af P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:21.003433 1207 raft_consensus.cc:383] T 321b9ad6d1ac4d1c80c7a01faf2015af P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:21.003748 1207 raft_consensus.cc:738] T 321b9ad6d1ac4d1c80c7a01faf2015af P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:21.004576 1207 consensus_queue.cc:260] T 321b9ad6d1ac4d1c80c7a01faf2015af P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:21.017658 1702 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55068 (request call id 154) took 1708 ms. Trace:
I20250114 02:54:21.018003 1702 rpcz_store.cc:276] 0114 02:54:19.309633 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:19.345877 (+ 36244us) service_pool.cc:225] Handling call
0114 02:54:21.012569 (+1666692us) ts_tablet_manager.cc:1145] Deleting tablet ed2d137579b8467da5eed4ea2f619a9a
0114 02:54:21.012614 (+ 45us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:21.012786 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1666118}
W20250114 02:54:21.019121 456 catalog_manager.cc:4839] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet ed2d137579b8467da5eed4ea2f619a9a because the tablet was not found. No further retry: Not found: Tablet not found: ed2d137579b8467da5eed4ea2f619a9a
I20250114 02:54:21.019402 1702 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55068 (request call id 155) took 1672 ms. Trace:
I20250114 02:54:21.019712 1702 rpcz_store.cc:276] 0114 02:54:19.347345 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:19.347497 (+ 152us) service_pool.cc:225] Handling call
0114 02:54:21.019147 (+1671650us) ts_tablet_manager.cc:1145] Deleting tablet 268a12df225e4ba59d4a84c2869d4661
0114 02:54:21.019189 (+ 42us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:21.019348 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1671148}
W20250114 02:54:21.020803 456 catalog_manager.cc:4839] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet 268a12df225e4ba59d4a84c2869d4661 because the tablet was not found. No further retry: Not found: Tablet not found: 268a12df225e4ba59d4a84c2869d4661
I20250114 02:54:21.023692 1702 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55068 (request call id 156) took 1649 ms. Trace:
I20250114 02:54:21.024039 1702 rpcz_store.cc:276] 0114 02:54:19.374630 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:19.374781 (+ 151us) service_pool.cc:225] Handling call
0114 02:54:21.023426 (+1648645us) ts_tablet_manager.cc:1145] Deleting tablet b63dd90f60a04465b948171e74888e79
0114 02:54:21.023475 (+ 49us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:21.023640 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1648080}
W20250114 02:54:21.025358 456 catalog_manager.cc:4839] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet b63dd90f60a04465b948171e74888e79 because the tablet was not found. No further retry: Not found: Tablet not found: b63dd90f60a04465b948171e74888e79
I20250114 02:54:21.025705 1702 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55068 (request call id 158) took 1623 ms. Trace:
I20250114 02:54:21.026010 1702 rpcz_store.cc:276] 0114 02:54:19.409152 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:19.426811 (+ 17659us) service_pool.cc:225] Handling call
0114 02:54:21.025482 (+1598671us) ts_tablet_manager.cc:1145] Deleting tablet c20aca761313498193e1d79c10705f8f
0114 02:54:21.025522 (+ 40us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:21.025659 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1598104}
I20250114 02:54:21.028726 1702 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55068 (request call id 159) took 1601 ms. Trace:
I20250114 02:54:21.029017 1702 rpcz_store.cc:276] 0114 02:54:19.427784 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:19.427948 (+ 164us) service_pool.cc:225] Handling call
0114 02:54:21.028465 (+1600517us) ts_tablet_manager.cc:1145] Deleting tablet 96b06fe4173343dabd726d253b9f9b5f
0114 02:54:21.028504 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:21.028677 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1599958}
I20250114 02:54:21.030417 463 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet a456420f78714cf895528e19dfe0ee8d
I20250114 02:54:21.031622 463 catalog_manager.cc:6596] ReplaceTablet: tablet a456420f78714cf895528e19dfe0ee8d deleted and replaced by tablet 670dc7ff15d644e18c296eeb2a6b99e1
I20250114 02:54:21.033123 466 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33540 (request call id 207) took 1178 ms. Trace:
I20250114 02:54:21.033429 1081 tablet_service.cc:1514] Processing DeleteTablet for tablet a456420f78714cf895528e19dfe0ee8d with delete_type TABLET_DATA_DELETED (replaced by tablet 670dc7ff15d644e18c296eeb2a6b99e1) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:21.033465 1702 tablet_replica.cc:331] T db24bf3db61941ba9b5e0662db23e748 P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:21.033429 466 rpcz_store.cc:276] 0114 02:54:19.855035 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:20.645913 (+790878us) service_pool.cc:225] Handling call
0114 02:54:20.709185 (+ 63272us) spinlock_profiling.cc:243] Waited 61.4 ms on lock 0x7b54000316d4. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616cad50 00007f646174df2a 00007f64633f2dec 00007f64634ec306 00007f6461250044 00007f646124ffd2 00007f646124ff01 00007f646124fe7c 00007f646124f132 00007f645f512dfc 00007f645f512236
0114 02:54:20.712151 (+ 2966us) catalog_manager.cc:2076] Acquired catalog manager lock
0114 02:54:20.771276 (+ 59125us) catalog_manager.cc:2122] Created new table and tablet info
0114 02:54:21.032520 (+261244us) catalog_manager.cc:2182] Wrote table and tablets to system table
0114 02:54:21.032824 (+ 304us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0114 02:54:21.033072 (+ 248us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:20.886782 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:20.887229 (+ 447us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:20.887259 (+ 30us) write_op.cc:435] Acquired schema lock
0114 02:54:20.887278 (+ 19us) tablet.cc:659] Decoding operations
0114 02:54:20.887635 (+ 357us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:20.887684 (+ 49us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:20.887717 (+ 33us) tablet.cc:682] Acquiring locks for 2 operations
0114 02:54:20.887983 (+ 266us) tablet.cc:698] Row locks acquired
0114 02:54:20.888001 (+ 18us) write_op.cc:260] PREPARE: finished
0114 02:54:20.888113 (+ 112us) write_op.cc:270] Start()
0114 02:54:20.888186 (+ 73us) write_op.cc:276] Timestamp: P: 1736823260888058 usec, L: 0
0114 02:54:20.888200 (+ 14us) op_driver.cc:348] REPLICATION: starting
0114 02:54:20.888643 (+ 443us) log.cc:844] Serialized 433 byte log entry
0114 02:54:21.025989 (+137346us) op_driver.cc:464] REPLICATION: finished
0114 02:54:21.027734 (+ 1745us) write_op.cc:301] APPLY: starting
0114 02:54:21.027787 (+ 53us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:21.027890 (+ 103us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:21.028017 (+ 127us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:21.028556 (+ 539us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:21.028703 (+ 147us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 02:54:21.028745 (+ 42us) write_op.cc:312] APPLY: finished
0114 02:54:21.028929 (+ 184us) log.cc:844] Serialized 29 byte log entry
0114 02:54:21.029072 (+ 143us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:21.029130 (+ 58us) write_op.cc:454] Released schema lock
0114 02:54:21.029164 (+ 34us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:21.029409 (+ 245us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:21.029433 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":141269248,"child_traces":[["op",{"apply.queue_time_us":1593,"apply.run_cpu_time_us":1602,"apply.run_wall_time_us":1602,"num_ops":2,"prepare.queue_time_us":114060,"prepare.run_cpu_time_us":2158,"prepare.run_wall_time_us":61575,"raft.queue_time_us":124,"raft.run_cpu_time_us":590,"raft.run_wall_time_us":591,"replication_time_us":137697}]]}
I20250114 02:54:21.036180 1207 ts_tablet_manager.cc:1428] T 321b9ad6d1ac4d1c80c7a01faf2015af P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.036s user 0.005s sys 0.000s
I20250114 02:54:21.034643 1702 raft_consensus.cc:2238] T db24bf3db61941ba9b5e0662db23e748 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:21.036962 789 tablet_service.cc:1514] Processing DeleteTablet for tablet a456420f78714cf895528e19dfe0ee8d with delete_type TABLET_DATA_DELETED (replaced by tablet 670dc7ff15d644e18c296eeb2a6b99e1) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:21.037665 1702 raft_consensus.cc:2267] T db24bf3db61941ba9b5e0662db23e748 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:21.039047 941 tablet_service.cc:1514] Processing DeleteTablet for tablet a456420f78714cf895528e19dfe0ee8d with delete_type TABLET_DATA_DELETED (replaced by tablet 670dc7ff15d644e18c296eeb2a6b99e1) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:21.040622 463 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_393cdd3f26b649d6b7b51051237e2c81" } new_table_name: "default.Table_f851d0fbfd4e4095912dbe3a96fd7142" modify_external_catalogs: true
I20250114 02:54:21.041446 466 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_1c87461bd6f64248aecf8fd2d9d3700b" } new_table_name: "default.Table_7058838b6bcb427786778096cf6f52b3" modify_external_catalogs: true
W20250114 02:54:21.038776 456 catalog_manager.cc:4839] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet c20aca761313498193e1d79c10705f8f because the tablet was not found. No further retry: Not found: Tablet not found: c20aca761313498193e1d79c10705f8f
W20250114 02:54:21.042363 456 catalog_manager.cc:4839] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet 96b06fe4173343dabd726d253b9f9b5f because the tablet was not found. No further retry: Not found: Tablet not found: 96b06fe4173343dabd726d253b9f9b5f
I20250114 02:54:21.043187 1702 ts_tablet_manager.cc:1905] T db24bf3db61941ba9b5e0662db23e748 P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:21.049602 1594 ts_tablet_manager.cc:1918] T 00080bea6c8e46e584ccba6dcb0e221b P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:21.050187 1594 log.cc:1198] T 00080bea6c8e46e584ccba6dcb0e221b P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/00080bea6c8e46e584ccba6dcb0e221b
I20250114 02:54:21.051007 1594 ts_tablet_manager.cc:1939] T 00080bea6c8e46e584ccba6dcb0e221b P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:21.051851 1594 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 138) took 1957 ms. Trace:
I20250114 02:54:21.052181 1594 rpcz_store.cc:276] 0114 02:54:19.094761 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:19.102612 (+ 7851us) service_pool.cc:225] Handling call
0114 02:54:20.322248 (+1219636us) ts_tablet_manager.cc:1145] Deleting tablet 00080bea6c8e46e584ccba6dcb0e221b
0114 02:54:20.322292 (+ 44us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:20.789873 (+467581us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:21.049306 (+259433us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:21.051802 (+ 2496us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1218931}
I20250114 02:54:21.057241 1207 tablet_bootstrap.cc:492] T 53a8bb487ad64157ad3baecf5d866eca P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:21.067489 1595 ts_tablet_manager.cc:1918] T ba215eb8422d4f879dd0bf5996543962 P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:21.067847 1595 log.cc:1198] T ba215eb8422d4f879dd0bf5996543962 P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/ba215eb8422d4f879dd0bf5996543962
I20250114 02:54:21.084899 1702 ts_tablet_manager.cc:1918] T db24bf3db61941ba9b5e0662db23e748 P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:21.085309 1702 log.cc:1198] T db24bf3db61941ba9b5e0662db23e748 P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/db24bf3db61941ba9b5e0662db23e748
I20250114 02:54:21.086059 1702 ts_tablet_manager.cc:1939] T db24bf3db61941ba9b5e0662db23e748 P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:21.089910 1207 tablet_bootstrap.cc:654] T 53a8bb487ad64157ad3baecf5d866eca P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:21.088104 1702 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55068 (request call id 160) took 1659 ms. Trace:
I20250114 02:54:21.093900 1702 rpcz_store.cc:276] 0114 02:54:19.429166 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:19.429307 (+ 141us) service_pool.cc:225] Handling call
0114 02:54:21.033307 (+1604000us) ts_tablet_manager.cc:1145] Deleting tablet db24bf3db61941ba9b5e0662db23e748
0114 02:54:21.033347 (+ 40us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:21.063838 (+ 30491us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:21.084644 (+ 20806us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:21.088042 (+ 3398us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1602966}
I20250114 02:54:21.095347 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet db24bf3db61941ba9b5e0662db23e748 (table default.Table_494541d472dc48d0bef33a634c1180fa [id=5ae72503278349c0bbe315174c896074]) successfully deleted
I20250114 02:54:21.095743 1702 tablet_replica.cc:331] T 99b8acb1fe594bf3825c7b9121018db6 P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:21.096709 1702 raft_consensus.cc:2238] T 99b8acb1fe594bf3825c7b9121018db6 P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Raft consensus shutting down.
I20250114 02:54:21.097172 1595 ts_tablet_manager.cc:1939] T ba215eb8422d4f879dd0bf5996543962 P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
I20250114 02:54:21.097481 1786 raft_consensus.cc:491] T 321b9ad6d1ac4d1c80c7a01faf2015af P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:21.097930 1702 pending_rounds.cc:62] T 99b8acb1fe594bf3825c7b9121018db6 P 525198593c8e47afaee54e74267b87e3: Trying to abort 1 pending ops.
I20250114 02:54:21.098201 1702 pending_rounds.cc:69] T 99b8acb1fe594bf3825c7b9121018db6 P 525198593c8e47afaee54e74267b87e3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114028067593228288 op_type: NO_OP noop_request { }
I20250114 02:54:21.097868 1786 raft_consensus.cc:513] T 321b9ad6d1ac4d1c80c7a01faf2015af P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:21.098608 1702 raft_consensus.cc:2883] T 99b8acb1fe594bf3825c7b9121018db6 P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250114 02:54:21.098929 1702 raft_consensus.cc:2267] T 99b8acb1fe594bf3825c7b9121018db6 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:21.099049 1595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 149) took 1899 ms. Trace:
I20250114 02:54:21.099354 1595 rpcz_store.cc:276] 0114 02:54:19.199170 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:19.199309 (+ 139us) service_pool.cc:225] Handling call
0114 02:54:20.966963 (+1767654us) ts_tablet_manager.cc:1145] Deleting tablet ba215eb8422d4f879dd0bf5996543962
0114 02:54:20.967002 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:21.011752 (+ 44750us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:21.067283 (+ 55531us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:21.098993 (+ 31710us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1767064}
I20250114 02:54:21.100050 1786 leader_election.cc:290] T 321b9ad6d1ac4d1c80c7a01faf2015af P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:21.100447 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet ba215eb8422d4f879dd0bf5996543962 (table default.Table_bacdf22532c84100a69a38cfa0ad07e5 [id=86304f6b48fd421cb953070ae7c9b0a2]) successfully deleted
I20250114 02:54:21.100826 1702 ts_tablet_manager.cc:1905] T 99b8acb1fe594bf3825c7b9121018db6 P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:21.103166 1101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321b9ad6d1ac4d1c80c7a01faf2015af" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:21.103863 1101 raft_consensus.cc:2463] T 321b9ad6d1ac4d1c80c7a01faf2015af P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:21.105065 898 leader_election.cc:304] T 321b9ad6d1ac4d1c80c7a01faf2015af P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:21.106105 1783 raft_consensus.cc:2798] T 321b9ad6d1ac4d1c80c7a01faf2015af P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:21.106403 1783 raft_consensus.cc:491] T 321b9ad6d1ac4d1c80c7a01faf2015af P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:21.106684 1783 raft_consensus.cc:3054] T 321b9ad6d1ac4d1c80c7a01faf2015af P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:21.107456 1595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 150) took 1908 ms. Trace:
I20250114 02:54:21.107760 1595 rpcz_store.cc:276] 0114 02:54:19.199487 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:19.199579 (+ 92us) service_pool.cc:225] Handling call
0114 02:54:21.107189 (+1907610us) ts_tablet_manager.cc:1145] Deleting tablet 268a12df225e4ba59d4a84c2869d4661
0114 02:54:21.107233 (+ 44us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:21.107398 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1907084}
I20250114 02:54:21.108815 1205 tablet_bootstrap.cc:492] T 66bfa92db9874750b3c340e31a06c412 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
W20250114 02:54:21.108886 458 catalog_manager.cc:4839] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 268a12df225e4ba59d4a84c2869d4661 because the tablet was not found. No further retry: Not found: Tablet not found: 268a12df225e4ba59d4a84c2869d4661
I20250114 02:54:21.109181 1205 ts_tablet_manager.cc:1397] T 66bfa92db9874750b3c340e31a06c412 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.166s user 0.012s sys 0.014s
I20250114 02:54:21.111272 1205 raft_consensus.cc:357] T 66bfa92db9874750b3c340e31a06c412 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:21.111840 1205 raft_consensus.cc:383] T 66bfa92db9874750b3c340e31a06c412 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:21.112048 1205 raft_consensus.cc:738] T 66bfa92db9874750b3c340e31a06c412 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:21.112659 1205 consensus_queue.cc:260] T 66bfa92db9874750b3c340e31a06c412 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:21.114413 1207 tablet_bootstrap.cc:492] T 53a8bb487ad64157ad3baecf5d866eca P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:21.114786 1207 ts_tablet_manager.cc:1397] T 53a8bb487ad64157ad3baecf5d866eca P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.058s user 0.004s sys 0.004s
I20250114 02:54:21.115327 1205 ts_tablet_manager.cc:1428] T 66bfa92db9874750b3c340e31a06c412 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250114 02:54:21.116956 1207 raft_consensus.cc:357] T 53a8bb487ad64157ad3baecf5d866eca P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:21.117568 1207 raft_consensus.cc:383] T 53a8bb487ad64157ad3baecf5d866eca P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:21.116959 1783 raft_consensus.cc:513] T 321b9ad6d1ac4d1c80c7a01faf2015af P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:21.117803 1207 raft_consensus.cc:738] T 53a8bb487ad64157ad3baecf5d866eca P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:21.118358 1207 consensus_queue.cc:260] T 53a8bb487ad64157ad3baecf5d866eca P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:21.121472 1595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 151) took 1921 ms. Trace:
I20250114 02:54:21.121711 1205 tablet_bootstrap.cc:492] T bc46815b4803418c95ea9c039a4fae88 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:21.121863 1595 rpcz_store.cc:276] 0114 02:54:19.200456 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:19.200542 (+ 86us) service_pool.cc:225] Handling call
0114 02:54:21.121215 (+1920673us) ts_tablet_manager.cc:1145] Deleting tablet ed2d137579b8467da5eed4ea2f619a9a
0114 02:54:21.121253 (+ 38us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:21.121415 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1920224}
W20250114 02:54:21.123322 458 catalog_manager.cc:4839] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet ed2d137579b8467da5eed4ea2f619a9a because the tablet was not found. No further retry: Not found: Tablet not found: ed2d137579b8467da5eed4ea2f619a9a
I20250114 02:54:21.126288 1101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321b9ad6d1ac4d1c80c7a01faf2015af" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:21.127036 1101 raft_consensus.cc:3054] T 321b9ad6d1ac4d1c80c7a01faf2015af P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:21.128577 1783 leader_election.cc:290] T 321b9ad6d1ac4d1c80c7a01faf2015af P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Requested vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:21.129559 1595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 152) took 1928 ms. Trace:
I20250114 02:54:21.129868 1595 rpcz_store.cc:276] 0114 02:54:19.201382 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:19.201471 (+ 89us) service_pool.cc:225] Handling call
0114 02:54:21.129328 (+1927857us) ts_tablet_manager.cc:1145] Deleting tablet 9589afd0094545c1801b7a0b5eb8698f
0114 02:54:21.129367 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:21.129511 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1927380}
W20250114 02:54:21.130941 458 catalog_manager.cc:4839] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 9589afd0094545c1801b7a0b5eb8698f because the tablet was not found. No further retry: Not found: Tablet not found: 9589afd0094545c1801b7a0b5eb8698f
I20250114 02:54:21.136996 1595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 153) took 1933 ms. Trace:
I20250114 02:54:21.137307 1595 rpcz_store.cc:276] 0114 02:54:19.203296 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:19.203392 (+ 96us) service_pool.cc:225] Handling call
0114 02:54:21.136755 (+1933363us) ts_tablet_manager.cc:1145] Deleting tablet c20aca761313498193e1d79c10705f8f
0114 02:54:21.136796 (+ 41us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:21.136947 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1932901}
W20250114 02:54:21.138386 458 catalog_manager.cc:4839] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet c20aca761313498193e1d79c10705f8f because the tablet was not found. No further retry: Not found: Tablet not found: c20aca761313498193e1d79c10705f8f
I20250114 02:54:21.141957 1595 tablet_replica.cc:331] T 7ecf782bb3d54528aa996abca1ea3a15 P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:21.142669 1595 raft_consensus.cc:2238] T 7ecf782bb3d54528aa996abca1ea3a15 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Raft consensus shutting down.
I20250114 02:54:21.143640 1205 tablet_bootstrap.cc:654] T bc46815b4803418c95ea9c039a4fae88 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:21.149231 1101 raft_consensus.cc:2463] T 321b9ad6d1ac4d1c80c7a01faf2015af P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:21.102067 816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321b9ad6d1ac4d1c80c7a01faf2015af" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:21.152638 816 raft_consensus.cc:2463] T 321b9ad6d1ac4d1c80c7a01faf2015af P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:21.119747 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321b9ad6d1ac4d1c80c7a01faf2015af" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:21.153569 818 raft_consensus.cc:3054] T 321b9ad6d1ac4d1c80c7a01faf2015af P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:21.154878 898 leader_election.cc:304] T 321b9ad6d1ac4d1c80c7a01faf2015af P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:21.155612 1783 raft_consensus.cc:2798] T 321b9ad6d1ac4d1c80c7a01faf2015af P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:21.156997 1783 raft_consensus.cc:695] T 321b9ad6d1ac4d1c80c7a01faf2015af P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Becoming Leader. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Running, Role: LEADER
I20250114 02:54:21.159421 1207 ts_tablet_manager.cc:1428] T 53a8bb487ad64157ad3baecf5d866eca P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.044s user 0.005s sys 0.001s
I20250114 02:54:21.164814 1595 raft_consensus.cc:2267] T 7ecf782bb3d54528aa996abca1ea3a15 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:21.165025 1702 ts_tablet_manager.cc:1918] T 99b8acb1fe594bf3825c7b9121018db6 P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:21.165954 818 raft_consensus.cc:2463] T 321b9ad6d1ac4d1c80c7a01faf2015af P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:21.166640 1595 ts_tablet_manager.cc:1905] T 7ecf782bb3d54528aa996abca1ea3a15 P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:21.167364 1702 log.cc:1198] T 99b8acb1fe594bf3825c7b9121018db6 P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/99b8acb1fe594bf3825c7b9121018db6
I20250114 02:54:21.017866 1206 tablet_bootstrap.cc:654] T 044832851f804208b0a371eb0857dbf3 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:21.168080 1702 ts_tablet_manager.cc:1939] T 99b8acb1fe594bf3825c7b9121018db6 P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:21.169947 1702 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55068 (request call id 161) took 1721 ms. Trace:
I20250114 02:54:21.170245 1702 rpcz_store.cc:276] 0114 02:54:19.448795 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:19.448932 (+ 137us) service_pool.cc:225] Handling call
0114 02:54:21.095556 (+1646624us) ts_tablet_manager.cc:1145] Deleting tablet 99b8acb1fe594bf3825c7b9121018db6
0114 02:54:21.095613 (+ 57us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:21.138622 (+ 43009us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:21.164811 (+ 26189us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:21.169884 (+ 5073us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1646110}
I20250114 02:54:21.171079 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet 00080bea6c8e46e584ccba6dcb0e221b (table default.Table_eb7878bdce82445c97f6ce665eb52d8e [id=3adecb85db3b44a4a65e50e61a91e908]) successfully deleted
I20250114 02:54:21.171247 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet 99b8acb1fe594bf3825c7b9121018db6 (table default.Table_661ed49bcbbb476aa68742b72604d95f [id=00383db95e10461a83394c5bd94ca94a]) successfully deleted
I20250114 02:54:21.172416 1205 tablet_bootstrap.cc:492] T bc46815b4803418c95ea9c039a4fae88 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:21.172536 1594 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 139) took 2029 ms. Trace:
I20250114 02:54:21.172791 1205 ts_tablet_manager.cc:1397] T bc46815b4803418c95ea9c039a4fae88 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.051s user 0.009s sys 0.001s
I20250114 02:54:21.172936 1594 rpcz_store.cc:276] 0114 02:54:19.143334 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:19.143442 (+ 108us) service_pool.cc:225] Handling call
0114 02:54:21.171936 (+2028494us) ts_tablet_manager.cc:1145] Deleting tablet b4740f6b3a6741368ba68907ad065279
0114 02:54:21.171979 (+ 43us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:21.172485 (+ 506us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2027901}
W20250114 02:54:21.174269 458 catalog_manager.cc:4839] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet b4740f6b3a6741368ba68907ad065279 because the tablet was not found. No further retry: Not found: Tablet not found: b4740f6b3a6741368ba68907ad065279
I20250114 02:54:21.175117 1205 raft_consensus.cc:357] T bc46815b4803418c95ea9c039a4fae88 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:21.175689 1594 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 142) took 2030 ms. Trace:
I20250114 02:54:21.175992 1205 raft_consensus.cc:383] T bc46815b4803418c95ea9c039a4fae88 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:21.176261 1205 raft_consensus.cc:738] T bc46815b4803418c95ea9c039a4fae88 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:21.176256 1594 rpcz_store.cc:276] 0114 02:54:19.145084 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:19.145171 (+ 87us) service_pool.cc:225] Handling call
0114 02:54:21.175403 (+2030232us) ts_tablet_manager.cc:1145] Deleting tablet 83b5b36a26e74668998a19fa0ddf7188
0114 02:54:21.175447 (+ 44us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:21.175617 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2029779}
I20250114 02:54:21.176846 1205 consensus_queue.cc:260] T bc46815b4803418c95ea9c039a4fae88 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
W20250114 02:54:21.177611 458 catalog_manager.cc:4839] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet 83b5b36a26e74668998a19fa0ddf7188 because the tablet was not found. No further retry: Not found: Tablet not found: 83b5b36a26e74668998a19fa0ddf7188
I20250114 02:54:21.178516 1207 tablet_bootstrap.cc:492] T 1381e26725174b9c877075cf5444f566 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:21.179562 1594 tablet_replica.cc:331] T 290ad07ab8be4791a39194909e000843 P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:21.181308 1783 consensus_queue.cc:237] T 321b9ad6d1ac4d1c80c7a01faf2015af P 525198593c8e47afaee54e74267b87e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:21.186700 1205 ts_tablet_manager.cc:1428] T bc46815b4803418c95ea9c039a4fae88 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20250114 02:54:21.187425 1702 tablet_replica.cc:331] T 961b6a25a13a4fa2a732a6383dcd8c84 P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:21.189906 1205 tablet_bootstrap.cc:492] T 266da813b4514329be0ad97d07b7114f P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:21.188155 1702 raft_consensus.cc:2238] T 961b6a25a13a4fa2a732a6383dcd8c84 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:21.194478 1702 raft_consensus.cc:2267] T 961b6a25a13a4fa2a732a6383dcd8c84 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:21.196404 1702 ts_tablet_manager.cc:1905] T 961b6a25a13a4fa2a732a6383dcd8c84 P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:21.204192 1205 tablet_bootstrap.cc:654] T 266da813b4514329be0ad97d07b7114f P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:21.206015 1786 raft_consensus.cc:491] T 1381e26725174b9c877075cf5444f566 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:21.206434 1786 raft_consensus.cc:513] T 1381e26725174b9c877075cf5444f566 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:21.208115 1786 leader_election.cc:290] T 1381e26725174b9c877075cf5444f566 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
W20250114 02:54:21.210003 455 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.125:46837, user_credentials={real_user=slave}} blocked reactor thread for 60650.7us
I20250114 02:54:21.212980 1101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1381e26725174b9c877075cf5444f566" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
W20250114 02:54:21.214206 898 leader_election.cc:343] T 1381e26725174b9c877075cf5444f566 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:21.229360 1207 tablet_bootstrap.cc:654] T 1381e26725174b9c877075cf5444f566 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:21.233605 1702 ts_tablet_manager.cc:1918] T 961b6a25a13a4fa2a732a6383dcd8c84 P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:21.233943 1702 log.cc:1198] T 961b6a25a13a4fa2a732a6383dcd8c84 P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/961b6a25a13a4fa2a732a6383dcd8c84
I20250114 02:54:21.233937 464 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_63cdf90ef15a453c9acb19962262eb35"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:21.234508 1702 ts_tablet_manager.cc:1939] T 961b6a25a13a4fa2a732a6383dcd8c84 P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
W20250114 02:54:21.235271 464 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_63cdf90ef15a453c9acb19962262eb35 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:21.245913 1207 tablet_bootstrap.cc:492] T 1381e26725174b9c877075cf5444f566 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:21.246307 1207 ts_tablet_manager.cc:1397] T 1381e26725174b9c877075cf5444f566 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.071s user 0.007s sys 0.001s
I20250114 02:54:21.210062 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1381e26725174b9c877075cf5444f566" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:21.247031 818 raft_consensus.cc:2463] T 1381e26725174b9c877075cf5444f566 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:21.247377 1594 raft_consensus.cc:2238] T 290ad07ab8be4791a39194909e000843 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:21.248303 898 leader_election.cc:304] T 1381e26725174b9c877075cf5444f566 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters: 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:21.264715 1207 raft_consensus.cc:357] T 1381e26725174b9c877075cf5444f566 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:21.265110 1783 raft_consensus.cc:2798] T 1381e26725174b9c877075cf5444f566 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:21.248307 1205 tablet_bootstrap.cc:492] T 266da813b4514329be0ad97d07b7114f P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:21.265393 1207 raft_consensus.cc:383] T 1381e26725174b9c877075cf5444f566 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:21.265436 1783 raft_consensus.cc:491] T 1381e26725174b9c877075cf5444f566 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:21.265653 1207 raft_consensus.cc:738] T 1381e26725174b9c877075cf5444f566 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:21.265691 1205 ts_tablet_manager.cc:1397] T 266da813b4514329be0ad97d07b7114f P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.076s user 0.006s sys 0.004s
I20250114 02:54:21.265990 1206 tablet_bootstrap.cc:492] T 044832851f804208b0a371eb0857dbf3 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:21.236181 1702 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55068 (request call id 162) took 1786 ms. Trace:
I20250114 02:54:21.266372 1206 ts_tablet_manager.cc:1397] T 044832851f804208b0a371eb0857dbf3 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.371s user 0.006s sys 0.003s
I20250114 02:54:21.265750 1783 raft_consensus.cc:3054] T 1381e26725174b9c877075cf5444f566 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:21.269315 1205 raft_consensus.cc:357] T 266da813b4514329be0ad97d07b7114f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:21.269383 1206 raft_consensus.cc:357] T 044832851f804208b0a371eb0857dbf3 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:21.269977 1205 raft_consensus.cc:383] T 266da813b4514329be0ad97d07b7114f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:21.270135 1206 raft_consensus.cc:383] T 044832851f804208b0a371eb0857dbf3 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:21.270238 1205 raft_consensus.cc:738] T 266da813b4514329be0ad97d07b7114f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:21.270371 1206 raft_consensus.cc:738] T 044832851f804208b0a371eb0857dbf3 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:21.270277 1207 consensus_queue.cc:260] T 1381e26725174b9c877075cf5444f566 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:21.270958 1205 consensus_queue.cc:260] T 266da813b4514329be0ad97d07b7114f P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:21.271348 1206 consensus_queue.cc:260] T 044832851f804208b0a371eb0857dbf3 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:21.247851 1594 raft_consensus.cc:2267] T 290ad07ab8be4791a39194909e000843 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:21.279215 1205 ts_tablet_manager.cc:1428] T 266da813b4514329be0ad97d07b7114f P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20250114 02:54:21.280989 1205 tablet_bootstrap.cc:492] T 26328f8e32bc49ed9b224f0e08b9f949 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:21.281842 1783 raft_consensus.cc:513] T 1381e26725174b9c877075cf5444f566 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:21.284282 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1381e26725174b9c877075cf5444f566" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:21.284795 818 raft_consensus.cc:3054] T 1381e26725174b9c877075cf5444f566 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:21.290241 1207 ts_tablet_manager.cc:1428] T 1381e26725174b9c877075cf5444f566 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.044s user 0.006s sys 0.001s
I20250114 02:54:21.290869 818 raft_consensus.cc:2463] T 1381e26725174b9c877075cf5444f566 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:21.266489 1702 rpcz_store.cc:276] 0114 02:54:19.450196 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:19.450335 (+ 139us) service_pool.cc:225] Handling call
0114 02:54:21.187279 (+1736944us) ts_tablet_manager.cc:1145] Deleting tablet 961b6a25a13a4fa2a732a6383dcd8c84
0114 02:54:21.187326 (+ 47us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:21.219043 (+ 31717us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:21.233402 (+ 14359us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:21.236117 (+ 2715us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1736429}
I20250114 02:54:21.291810 898 leader_election.cc:304] T 1381e26725174b9c877075cf5444f566 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
I20250114 02:54:21.292387 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet 961b6a25a13a4fa2a732a6383dcd8c84 (table default.Table_0943b1ca347042c9a6043006bc566701 [id=e3418787ef0a4e22979fb41d4f94ebb0]) successfully deleted
I20250114 02:54:21.292666 1786 raft_consensus.cc:2798] T 1381e26725174b9c877075cf5444f566 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election won for term 1
W20250114 02:54:21.293692 466 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 84738a6c78b24c1fb64013048763b18f consensus state has no leader
I20250114 02:54:21.299741 1702 tablet_replica.cc:331] T 266da813b4514329be0ad97d07b7114f P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:21.300659 1702 raft_consensus.cc:2238] T 266da813b4514329be0ad97d07b7114f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:21.301049 1702 raft_consensus.cc:2267] T 266da813b4514329be0ad97d07b7114f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:21.302907 1205 tablet_bootstrap.cc:654] T 26328f8e32bc49ed9b224f0e08b9f949 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:21.301855 1101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1381e26725174b9c877075cf5444f566" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:21.304759 1101 raft_consensus.cc:3054] T 1381e26725174b9c877075cf5444f566 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:21.306035 1702 ts_tablet_manager.cc:1905] T 266da813b4514329be0ad97d07b7114f P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:21.311173 1101 raft_consensus.cc:2463] T 1381e26725174b9c877075cf5444f566 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:21.315711 1783 leader_election.cc:290] T 1381e26725174b9c877075cf5444f566 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Requested vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:21.324472 1207 tablet_bootstrap.cc:492] T 266da813b4514329be0ad97d07b7114f P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:21.329212 1595 ts_tablet_manager.cc:1918] T 7ecf782bb3d54528aa996abca1ea3a15 P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:21.329579 1595 log.cc:1198] T 7ecf782bb3d54528aa996abca1ea3a15 P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/7ecf782bb3d54528aa996abca1ea3a15
I20250114 02:54:21.330269 1595 ts_tablet_manager.cc:1939] T 7ecf782bb3d54528aa996abca1ea3a15 P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
I20250114 02:54:21.330271 1205 tablet_bootstrap.cc:492] T 26328f8e32bc49ed9b224f0e08b9f949 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:21.330724 1205 ts_tablet_manager.cc:1397] T 26328f8e32bc49ed9b224f0e08b9f949 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.050s user 0.008s sys 0.002s
I20250114 02:54:21.332974 468 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet f464eabfbc0945fa8a5583f00c1dab40 (table default.Table_4fd891b63924429085d78f78cb177f0e [id=d36c71e7562b4ff1bc8be5cf2955000e]). Expected version 1 got 0
I20250114 02:54:21.333391 1205 raft_consensus.cc:357] T 26328f8e32bc49ed9b224f0e08b9f949 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:21.333807 468 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet aad96f2507974023aaa988e6950feb7a (table default.Table_9676926a50eb42349a2d609b30566e9a [id=f38063ebf94d4084ab0174428503dc29]). Expected version 2 got 0
I20250114 02:54:21.334064 1205 raft_consensus.cc:383] T 26328f8e32bc49ed9b224f0e08b9f949 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:21.334339 1205 raft_consensus.cc:738] T 26328f8e32bc49ed9b224f0e08b9f949 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:21.334985 1205 consensus_queue.cc:260] T 26328f8e32bc49ed9b224f0e08b9f949 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:21.341410 462 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet aad96f2507974023aaa988e6950feb7a (table default.Table_9676926a50eb42349a2d609b30566e9a [id=f38063ebf94d4084ab0174428503dc29]). Expected version 2 got 0
I20250114 02:54:21.342307 462 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 55b68586ef8b45439a7dd8dcd2efcd79 (table default.Table_66f2263162f64250a0b49073abe4bd4c [id=c73ede1a5f664d8eb204fb9a012f8f7a]). Expected version 1 got 0
I20250114 02:54:21.343168 462 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 1381e26725174b9c877075cf5444f566 (table default.Table_b624816662024c55a5f5ef5830b20b7c [id=d2ea84ea59ad4b03ad16bb3052a441f4]). Expected version 2 got 0
I20250114 02:54:21.332116 1595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 154) took 2127 ms. Trace:
I20250114 02:54:21.345636 1595 rpcz_store.cc:276] 0114 02:54:19.205058 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:19.205144 (+ 86us) service_pool.cc:225] Handling call
0114 02:54:21.141821 (+1936677us) ts_tablet_manager.cc:1145] Deleting tablet 7ecf782bb3d54528aa996abca1ea3a15
0114 02:54:21.141857 (+ 36us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:21.206346 (+ 64489us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:21.328967 (+122621us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:21.332047 (+ 3080us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21299,"tablet-delete.queue_time_us":1936243}
I20250114 02:54:21.346935 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet 7ecf782bb3d54528aa996abca1ea3a15 (table default.Table_b0cd93e5a498450e84fb0f1ba4be4505 [id=c70751b59bfc4c6fa60d0e8bb40d5f36]) successfully deleted
I20250114 02:54:21.348713 1786 raft_consensus.cc:695] T 1381e26725174b9c877075cf5444f566 P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Becoming Leader. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Running, Role: LEADER
I20250114 02:54:21.349586 1786 consensus_queue.cc:237] T 1381e26725174b9c877075cf5444f566 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:21.351969 941 tablet_service.cc:1467] Processing CreateTablet for tablet 967f4d5b944543d49e87645d069b5b55 (DEFAULT_TABLE table=default.Table_1ec80cfdcee64e98beccc30fec0afff8 [id=9cf0c5a5c3434e348bb3d159e9638924]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:21.352916 941 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 967f4d5b944543d49e87645d069b5b55. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:21.352952 1081 tablet_service.cc:1467] Processing CreateTablet for tablet 967f4d5b944543d49e87645d069b5b55 (DEFAULT_TABLE table=default.Table_1ec80cfdcee64e98beccc30fec0afff8 [id=9cf0c5a5c3434e348bb3d159e9638924]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:21.353580 1205 ts_tablet_manager.cc:1428] T 26328f8e32bc49ed9b224f0e08b9f949 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.023s user 0.006s sys 0.003s
I20250114 02:54:21.353873 1081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 967f4d5b944543d49e87645d069b5b55. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:21.355710 936 tablet_service.cc:1467] Processing CreateTablet for tablet 141cede72b8741a8b0615d7c6bb823bb (DEFAULT_TABLE table=default.Table_80da487b7f04491888501948c55ed09f [id=13fc996d911b430290e37a94dd44b1ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:21.356552 936 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 141cede72b8741a8b0615d7c6bb823bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:21.356922 939 tablet_service.cc:1467] Processing CreateTablet for tablet 30a56fb2d941470590e17f37b90aef26 (DEFAULT_TABLE table=default.Table_1ebcd922bb294485b52278a74830aaab [id=66082998119e4c43a666fb43087fc622]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:21.357776 939 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30a56fb2d941470590e17f37b90aef26. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:21.343995 462 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet f464eabfbc0945fa8a5583f00c1dab40 (table default.Table_4fd891b63924429085d78f78cb177f0e [id=d36c71e7562b4ff1bc8be5cf2955000e]). Expected version 1 got 0
I20250114 02:54:21.359308 1084 tablet_service.cc:1467] Processing CreateTablet for tablet 30a56fb2d941470590e17f37b90aef26 (DEFAULT_TABLE table=default.Table_1ebcd922bb294485b52278a74830aaab [id=66082998119e4c43a666fb43087fc622]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:21.360205 1084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30a56fb2d941470590e17f37b90aef26. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:21.361054 1083 tablet_service.cc:1467] Processing CreateTablet for tablet 141cede72b8741a8b0615d7c6bb823bb (DEFAULT_TABLE table=default.Table_80da487b7f04491888501948c55ed09f [id=13fc996d911b430290e37a94dd44b1ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:21.361872 1083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 141cede72b8741a8b0615d7c6bb823bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:21.365276 465 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet aad96f2507974023aaa988e6950feb7a (table default.Table_9676926a50eb42349a2d609b30566e9a [id=f38063ebf94d4084ab0174428503dc29]). Expected version 2 got 0
I20250114 02:54:21.367159 933 tablet_service.cc:1467] Processing CreateTablet for tablet f40c66314342453b898da3ac0e7e86bf (DEFAULT_TABLE table=default.Table_bacdf22532c84100a69a38cfa0ad07e5 [id=86304f6b48fd421cb953070ae7c9b0a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:21.368281 933 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f40c66314342453b898da3ac0e7e86bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:21.380434 1207 tablet_bootstrap.cc:654] T 266da813b4514329be0ad97d07b7114f P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:21.382745 788 tablet_service.cc:1467] Processing CreateTablet for tablet 30a56fb2d941470590e17f37b90aef26 (DEFAULT_TABLE table=default.Table_1ebcd922bb294485b52278a74830aaab [id=66082998119e4c43a666fb43087fc622]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:21.383473 1784 raft_consensus.cc:491] T 53a8bb487ad64157ad3baecf5d866eca P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:21.383770 788 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30a56fb2d941470590e17f37b90aef26. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:21.383893 1784 raft_consensus.cc:513] T 53a8bb487ad64157ad3baecf5d866eca P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:21.385368 789 tablet_service.cc:1467] Processing CreateTablet for tablet 967f4d5b944543d49e87645d069b5b55 (DEFAULT_TABLE table=default.Table_1ec80cfdcee64e98beccc30fec0afff8 [id=9cf0c5a5c3434e348bb3d159e9638924]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:21.385762 1784 leader_election.cc:290] T 53a8bb487ad64157ad3baecf5d866eca P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:21.386363 789 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 967f4d5b944543d49e87645d069b5b55. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:21.388013 959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53a8bb487ad64157ad3baecf5d866eca" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:21.388624 959 raft_consensus.cc:2463] T 53a8bb487ad64157ad3baecf5d866eca P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 0.
I20250114 02:54:21.388832 796 tablet_service.cc:1467] Processing CreateTablet for tablet 141cede72b8741a8b0615d7c6bb823bb (DEFAULT_TABLE table=default.Table_80da487b7f04491888501948c55ed09f [id=13fc996d911b430290e37a94dd44b1ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:21.389673 796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 141cede72b8741a8b0615d7c6bb823bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:21.390149 1594 ts_tablet_manager.cc:1905] T 290ad07ab8be4791a39194909e000843 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:21.391503 1040 leader_election.cc:304] T 53a8bb487ad64157ad3baecf5d866eca P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:21.392280 1784 raft_consensus.cc:2798] T 53a8bb487ad64157ad3baecf5d866eca P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:21.392585 1784 raft_consensus.cc:491] T 53a8bb487ad64157ad3baecf5d866eca P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:21.392688 1652 raft_consensus.cc:491] T 044832851f804208b0a371eb0857dbf3 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:21.392863 1784 raft_consensus.cc:3054] T 53a8bb487ad64157ad3baecf5d866eca P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:21.392925 790 tablet_service.cc:1467] Processing CreateTablet for tablet f40c66314342453b898da3ac0e7e86bf (DEFAULT_TABLE table=default.Table_bacdf22532c84100a69a38cfa0ad07e5 [id=86304f6b48fd421cb953070ae7c9b0a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:21.393090 1652 raft_consensus.cc:513] T 044832851f804208b0a371eb0857dbf3 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:21.393883 790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f40c66314342453b898da3ac0e7e86bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:21.395464 1652 leader_election.cc:290] T 044832851f804208b0a371eb0857dbf3 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:21.397399 1099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "044832851f804208b0a371eb0857dbf3" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:21.397544 959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "044832851f804208b0a371eb0857dbf3" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
W20250114 02:54:21.397945 1042 leader_election.cc:343] T 53a8bb487ad64157ad3baecf5d866eca P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Not found: Tablet not found: 53a8bb487ad64157ad3baecf5d866eca
W20250114 02:54:21.398713 753 leader_election.cc:343] T 044832851f804208b0a371eb0857dbf3 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
W20250114 02:54:21.399263 751 leader_election.cc:343] T 044832851f804208b0a371eb0857dbf3 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:21.399592 751 leader_election.cc:304] T 044832851f804208b0a371eb0857dbf3 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:21.402815 1652 raft_consensus.cc:2743] T 044832851f804208b0a371eb0857dbf3 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:21.412792 1784 raft_consensus.cc:513] T 53a8bb487ad64157ad3baecf5d866eca P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:21.413882 1206 ts_tablet_manager.cc:1428] T 044832851f804208b0a371eb0857dbf3 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.147s user 0.007s sys 0.001s
I20250114 02:54:21.415568 1205 tablet_bootstrap.cc:492] T 55b68586ef8b45439a7dd8dcd2efcd79 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:21.416065 1797 raft_consensus.cc:491] T 55b68586ef8b45439a7dd8dcd2efcd79 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:21.416491 1797 raft_consensus.cc:513] T 55b68586ef8b45439a7dd8dcd2efcd79 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:21.414893 1206 tablet_bootstrap.cc:492] T 0304df55431e42e1a999ed17dc35e6c0 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:21.418339 1797 leader_election.cc:290] T 55b68586ef8b45439a7dd8dcd2efcd79 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
W20250114 02:54:21.420073 1042 leader_election.cc:343] T 53a8bb487ad64157ad3baecf5d866eca P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Not found: Tablet not found: 53a8bb487ad64157ad3baecf5d866eca
I20250114 02:54:21.421391 959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55b68586ef8b45439a7dd8dcd2efcd79" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
W20250114 02:54:21.422595 1040 leader_election.cc:343] T 55b68586ef8b45439a7dd8dcd2efcd79 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:21.423305 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55b68586ef8b45439a7dd8dcd2efcd79" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:21.423615 1702 ts_tablet_manager.cc:1918] T 266da813b4514329be0ad97d07b7114f P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:21.423960 1702 log.cc:1198] T 266da813b4514329be0ad97d07b7114f P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/266da813b4514329be0ad97d07b7114f
I20250114 02:54:21.423887 818 raft_consensus.cc:2463] T 55b68586ef8b45439a7dd8dcd2efcd79 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 0.
I20250114 02:54:21.424377 1595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 159) took 1648 ms. Trace:
I20250114 02:54:21.424899 1042 leader_election.cc:304] T 55b68586ef8b45439a7dd8dcd2efcd79 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters: 525198593c8e47afaee54e74267b87e3
I20250114 02:54:21.426079 1797 raft_consensus.cc:2798] T 55b68586ef8b45439a7dd8dcd2efcd79 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:21.426415 1797 raft_consensus.cc:491] T 55b68586ef8b45439a7dd8dcd2efcd79 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:21.426707 1797 raft_consensus.cc:3054] T 55b68586ef8b45439a7dd8dcd2efcd79 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:21.443935 1784 leader_election.cc:290] T 53a8bb487ad64157ad3baecf5d866eca P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Requested vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:21.444808 1702 ts_tablet_manager.cc:1939] T 266da813b4514329be0ad97d07b7114f P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:21.444952 959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53a8bb487ad64157ad3baecf5d866eca" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
I20250114 02:54:21.445474 959 raft_consensus.cc:3054] T 53a8bb487ad64157ad3baecf5d866eca P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:21.445539 1702 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55068 (request call id 167) took 1654 ms. Trace:
I20250114 02:54:21.445833 1702 rpcz_store.cc:276] 0114 02:54:19.791364 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:20.016618 (+225254us) service_pool.cc:225] Handling call
0114 02:54:21.299590 (+1282972us) ts_tablet_manager.cc:1145] Deleting tablet 266da813b4514329be0ad97d07b7114f
0114 02:54:21.299635 (+ 45us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:21.322439 (+ 22804us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:21.423416 (+100977us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:21.445491 (+ 22075us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1223023}
I20250114 02:54:21.446822 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet 266da813b4514329be0ad97d07b7114f (table default.Table_ae220acb7e0345bdbb80b11398e4d153 [id=bcf9e834733d49a8bc235cd303a75a35]) successfully deleted
I20250114 02:54:21.426138 1595 rpcz_store.cc:276] 0114 02:54:19.775568 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:19.775706 (+ 138us) service_pool.cc:225] Handling call
0114 02:54:21.423582 (+1647876us) ts_tablet_manager.cc:1145] Deleting tablet 266da813b4514329be0ad97d07b7114f
0114 02:54:21.423652 (+ 70us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:21.424296 (+ 644us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1647351}
W20250114 02:54:21.451088 458 catalog_manager.cc:4852] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 266da813b4514329be0ad97d07b7114f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 266da813b4514329be0ad97d07b7114f already in progress: creating tablet
I20250114 02:54:21.452437 959 raft_consensus.cc:2463] T 53a8bb487ad64157ad3baecf5d866eca P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1.
I20250114 02:54:21.443137 1797 raft_consensus.cc:513] T 55b68586ef8b45439a7dd8dcd2efcd79 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:21.456944 1702 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55068 (request call id 179) took 1091 ms. Trace:
I20250114 02:54:21.457095 1595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 171) took 1091 ms. Trace:
I20250114 02:54:21.457379 1595 rpcz_store.cc:276] 0114 02:54:20.365748 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:20.399663 (+ 33915us) service_pool.cc:225] Handling call
0114 02:54:21.456772 (+1057109us) ts_tablet_manager.cc:1145] Deleting tablet 46360f726fad4534a2daf67581346fc1
0114 02:54:21.456811 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:21.457035 (+ 224us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1054851}
I20250114 02:54:21.458016 1205 tablet_bootstrap.cc:654] T 55b68586ef8b45439a7dd8dcd2efcd79 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:21.459113 1595 tablet_replica.cc:331] T a456420f78714cf895528e19dfe0ee8d P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:21.459779 1595 raft_consensus.cc:2238] T a456420f78714cf895528e19dfe0ee8d P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:21.459811 961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55b68586ef8b45439a7dd8dcd2efcd79" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
W20250114 02:54:21.461185 1040 leader_election.cc:343] T 55b68586ef8b45439a7dd8dcd2efcd79 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
W20250114 02:54:21.461264 458 catalog_manager.cc:4852] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 46360f726fad4534a2daf67581346fc1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46360f726fad4534a2daf67581346fc1 already in progress: creating tablet
I20250114 02:54:21.462200 1040 leader_election.cc:304] T 53a8bb487ad64157ad3baecf5d866eca P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters: 819f9e3412e54d44abb11d8b3ca877c2
W20250114 02:54:21.462822 458 catalog_manager.cc:4852] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet ca986f81ed6848a89588366f3f400eec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca986f81ed6848a89588366f3f400eec already in progress: creating tablet
I20250114 02:54:21.463158 1784 raft_consensus.cc:2798] T 53a8bb487ad64157ad3baecf5d866eca P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:21.463847 1797 leader_election.cc:290] T 55b68586ef8b45439a7dd8dcd2efcd79 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Requested vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:21.367956 465 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet f464eabfbc0945fa8a5583f00c1dab40 (table default.Table_4fd891b63924429085d78f78cb177f0e [id=d36c71e7562b4ff1bc8be5cf2955000e]). Expected version 1 got 0
I20250114 02:54:21.472450 1784 raft_consensus.cc:695] T 53a8bb487ad64157ad3baecf5d866eca P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Becoming Leader. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Running, Role: LEADER
I20250114 02:54:21.473489 1784 consensus_queue.cc:237] T 53a8bb487ad64157ad3baecf5d866eca P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:21.475924 1207 tablet_bootstrap.cc:492] T 266da813b4514329be0ad97d07b7114f P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:21.476325 1207 ts_tablet_manager.cc:1397] T 266da813b4514329be0ad97d07b7114f P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.152s user 0.007s sys 0.003s
I20250114 02:54:21.457234 1702 rpcz_store.cc:276] 0114 02:54:20.365484 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:20.369688 (+ 4204us) service_pool.cc:225] Handling call
0114 02:54:21.456634 (+1086946us) ts_tablet_manager.cc:1145] Deleting tablet 46360f726fad4534a2daf67581346fc1
0114 02:54:21.456672 (+ 38us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:21.456893 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1085543}
I20250114 02:54:21.477553 1702 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55068 (request call id 180) took 1007 ms. Trace:
W20250114 02:54:21.478142 456 catalog_manager.cc:4852] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet 46360f726fad4534a2daf67581346fc1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46360f726fad4534a2daf67581346fc1 already in progress: creating tablet
I20250114 02:54:21.478601 1207 raft_consensus.cc:357] T 266da813b4514329be0ad97d07b7114f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:21.479244 1207 raft_consensus.cc:383] T 266da813b4514329be0ad97d07b7114f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:21.477854 1702 rpcz_store.cc:276] 0114 02:54:20.469813 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:20.469923 (+ 110us) service_pool.cc:225] Handling call
0114 02:54:21.477202 (+1007279us) ts_tablet_manager.cc:1145] Deleting tablet ca986f81ed6848a89588366f3f400eec
0114 02:54:21.477241 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:21.477503 (+ 262us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1006799}
I20250114 02:54:21.479496 1207 raft_consensus.cc:738] T 266da813b4514329be0ad97d07b7114f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
W20250114 02:54:21.480125 456 catalog_manager.cc:4852] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet ca986f81ed6848a89588366f3f400eec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca986f81ed6848a89588366f3f400eec already in progress: creating tablet
I20250114 02:54:21.481303 1702 tablet_replica.cc:331] T a456420f78714cf895528e19dfe0ee8d P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:21.486053 1595 raft_consensus.cc:2267] T a456420f78714cf895528e19dfe0ee8d P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:21.487968 1595 ts_tablet_manager.cc:1905] T a456420f78714cf895528e19dfe0ee8d P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:21.489816 466 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet aad96f2507974023aaa988e6950feb7a (table default.Table_9676926a50eb42349a2d609b30566e9a [id=f38063ebf94d4084ab0174428503dc29]). Expected version 2 got 0
I20250114 02:54:21.490331 1702 raft_consensus.cc:2238] T a456420f78714cf895528e19dfe0ee8d P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:21.490797 1702 raft_consensus.cc:2267] T a456420f78714cf895528e19dfe0ee8d P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:21.505208 1081 tablet_service.cc:1467] Processing CreateTablet for tablet f40c66314342453b898da3ac0e7e86bf (DEFAULT_TABLE table=default.Table_bacdf22532c84100a69a38cfa0ad07e5 [id=86304f6b48fd421cb953070ae7c9b0a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:21.505422 1084 tablet_service.cc:1467] Processing CreateTablet for tablet 0b8989d340fa4ae992ef97e226d41110 (DEFAULT_TABLE table=default.Table_24cfc302131c4035b4c8e0fdc03625fa [id=d772a301194a4e9ab90cabfc05104081]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:21.506290 1081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f40c66314342453b898da3ac0e7e86bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:21.506866 1084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b8989d340fa4ae992ef97e226d41110. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:21.516044 936 tablet_service.cc:1467] Processing CreateTablet for tablet d2a7b2bee5c1440fb705b172c18a9247 (DEFAULT_TABLE table=default.Table_910b0ec1594a4dfcbb342055cab3eb65 [id=4d2f7fabf8a44155a586a2b79a8fca53]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:21.517053 936 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2a7b2bee5c1440fb705b172c18a9247. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:21.517266 1783 raft_consensus.cc:491] T bc46815b4803418c95ea9c039a4fae88 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:21.517390 1083 tablet_service.cc:1467] Processing CreateTablet for tablet d2a7b2bee5c1440fb705b172c18a9247 (DEFAULT_TABLE table=default.Table_910b0ec1594a4dfcbb342055cab3eb65 [id=4d2f7fabf8a44155a586a2b79a8fca53]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:21.517684 1783 raft_consensus.cc:513] T bc46815b4803418c95ea9c039a4fae88 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:21.518280 1083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2a7b2bee5c1440fb705b172c18a9247. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:21.519519 1206 tablet_bootstrap.cc:654] T 0304df55431e42e1a999ed17dc35e6c0 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:21.519502 1783 leader_election.cc:290] T bc46815b4803418c95ea9c039a4fae88 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:21.520988 1099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc46815b4803418c95ea9c039a4fae88" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
W20250114 02:54:21.522142 898 leader_election.cc:343] T bc46815b4803418c95ea9c039a4fae88 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:21.524485 933 tablet_service.cc:1467] Processing CreateTablet for tablet 0b8989d340fa4ae992ef97e226d41110 (DEFAULT_TABLE table=default.Table_24cfc302131c4035b4c8e0fdc03625fa [id=d772a301194a4e9ab90cabfc05104081]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:21.525105 466 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet 55b68586ef8b45439a7dd8dcd2efcd79 (table default.Table_66f2263162f64250a0b49073abe4bd4c [id=c73ede1a5f664d8eb204fb9a012f8f7a]). Expected version 1 got 0
I20250114 02:54:21.480037 1207 consensus_queue.cc:260] T 266da813b4514329be0ad97d07b7114f P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:21.525676 933 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b8989d340fa4ae992ef97e226d41110. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:21.526005 466 catalog_manager.cc:5526] T 26328f8e32bc49ed9b224f0e08b9f949 P 90d6c2c58bb54e0eafb365625ac33871 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67). New cstate: current_term: 1 leader_uuid: "90d6c2c58bb54e0eafb365625ac33871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } }
I20250114 02:54:21.527356 1207 ts_tablet_manager.cc:1428] T 266da813b4514329be0ad97d07b7114f P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.051s user 0.005s sys 0.000s
I20250114 02:54:21.527829 466 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet f464eabfbc0945fa8a5583f00c1dab40 (table default.Table_4fd891b63924429085d78f78cb177f0e [id=d36c71e7562b4ff1bc8be5cf2955000e]). Expected version 1 got 0
I20250114 02:54:21.536363 470 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet ec3285525faf482d8739f1bedd653503
I20250114 02:54:21.537927 1086 tablet_service.cc:1514] Processing DeleteTablet for tablet ec3285525faf482d8739f1bedd653503 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:20 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:21.537963 470 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:33540 (request call id 209) took 1561 ms. Trace:
I20250114 02:54:21.540050 1702 ts_tablet_manager.cc:1905] T a456420f78714cf895528e19dfe0ee8d P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:21.541585 941 tablet_service.cc:1514] Processing DeleteTablet for tablet ec3285525faf482d8739f1bedd653503 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:20 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:21.538332 470 rpcz_store.cc:276] 0114 02:54:19.976745 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:20.812570 (+835825us) service_pool.cc:225] Handling call
0114 02:54:20.823074 (+ 10504us) spinlock_profiling.cc:243] Waited 9.27 ms on lock 0x7b54000316d4. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616cad50 00007f646174df2a 00007f64633f2dec 00007f64634ec616 00007f6461254b44 00007f6461254ad2 00007f6461254a01 00007f646125497c 00007f6461253c32 00007f645f512dfc 00007f645f512236
0114 02:54:20.823968 (+ 894us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0114 02:54:20.827956 (+ 3988us) catalog_manager.cc:2753] Modifying in-memory table state
0114 02:54:20.828108 (+ 152us) catalog_manager.cc:2762] Locking tablets
0114 02:54:20.857616 (+ 29508us) catalog_manager.cc:2776] Removing table and tablets from system table
0114 02:54:21.032157 (+174541us) catalog_manager.cc:2794] Removing table from by-name map
0114 02:54:21.032403 (+ 246us) catalog_manager.cc:2810] Committing in-memory state
0114 02:54:21.536273 (+503870us) catalog_manager.cc:2814] Aborting table tasks
0114 02:54:21.537894 (+ 1621us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:20.952398 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:20.952810 (+ 412us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:20.952840 (+ 30us) write_op.cc:435] Acquired schema lock
0114 02:54:20.952859 (+ 19us) tablet.cc:659] Decoding operations
0114 02:54:20.953209 (+ 350us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:20.953249 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:20.953279 (+ 30us) tablet.cc:682] Acquiring locks for 2 operations
0114 02:54:20.953570 (+ 291us) tablet.cc:698] Row locks acquired
0114 02:54:20.953587 (+ 17us) write_op.cc:260] PREPARE: finished
0114 02:54:20.953702 (+ 115us) write_op.cc:270] Start()
0114 02:54:20.953782 (+ 80us) write_op.cc:276] Timestamp: P: 1736823260953633 usec, L: 0
0114 02:54:20.953796 (+ 14us) op_driver.cc:348] REPLICATION: starting
0114 02:54:20.954218 (+ 422us) log.cc:844] Serialized 762 byte log entry
0114 02:54:21.027334 (+ 73116us) op_driver.cc:464] REPLICATION: finished
0114 02:54:21.029657 (+ 2323us) write_op.cc:301] APPLY: starting
0114 02:54:21.029701 (+ 44us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:21.029790 (+ 89us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:21.029890 (+ 100us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:21.030115 (+ 225us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:21.030232 (+ 117us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0114 02:54:21.030271 (+ 39us) write_op.cc:312] APPLY: finished
0114 02:54:21.030435 (+ 164us) log.cc:844] Serialized 29 byte log entry
0114 02:54:21.030554 (+ 119us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:21.030612 (+ 58us) write_op.cc:454] Released schema lock
0114 02:54:21.030639 (+ 27us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:21.047254 (+ 16615us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:21.047284 (+ 30us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":21322624,"child_traces":[["op",{"apply.queue_time_us":2209,"apply.run_cpu_time_us":1143,"apply.run_wall_time_us":1143,"num_ops":2,"prepare.queue_time_us":92781,"prepare.run_cpu_time_us":2219,"prepare.run_wall_time_us":94866,"raft.queue_time_us":142,"raft.run_cpu_time_us":305,"raft.run_wall_time_us":305,"replication_time_us":73472,"thread_start_us":11745,"threads_started":1}]]}
I20250114 02:54:21.543439 1594 ts_tablet_manager.cc:1918] T 290ad07ab8be4791a39194909e000843 P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:21.543790 1594 log.cc:1198] T 290ad07ab8be4791a39194909e000843 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/290ad07ab8be4791a39194909e000843
I20250114 02:54:21.544544 470 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_78eb79361f2c42538d83e9edcf6b51dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:21.545931 470 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_78eb79361f2c42538d83e9edcf6b51dc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:21.550218 1207 tablet_bootstrap.cc:492] T bc46815b4803418c95ea9c039a4fae88 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:21.557905 1205 tablet_bootstrap.cc:492] T 55b68586ef8b45439a7dd8dcd2efcd79 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:21.558347 1205 ts_tablet_manager.cc:1397] T 55b68586ef8b45439a7dd8dcd2efcd79 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.143s user 0.008s sys 0.008s
I20250114 02:54:21.568077 470 master_service.cc:946] ReplaceTablet: received request to replace tablet 6e858a8449444ef999b9bbef612249d3 from {username='slave'} at 127.0.0.1:33540
I20250114 02:54:21.586462 1205 raft_consensus.cc:357] T 55b68586ef8b45439a7dd8dcd2efcd79 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:21.587154 1205 raft_consensus.cc:383] T 55b68586ef8b45439a7dd8dcd2efcd79 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:21.587411 1205 raft_consensus.cc:738] T 55b68586ef8b45439a7dd8dcd2efcd79 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:21.597395 464 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet aad96f2507974023aaa988e6950feb7a (table default.Table_9676926a50eb42349a2d609b30566e9a [id=f38063ebf94d4084ab0174428503dc29]). Expected version 2 got 0
I20250114 02:54:21.598357 464 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 55b68586ef8b45439a7dd8dcd2efcd79 (table default.Table_66f2263162f64250a0b49073abe4bd4c [id=c73ede1a5f664d8eb204fb9a012f8f7a]). Expected version 1 got 0
I20250114 02:54:21.588104 1205 consensus_queue.cc:260] T 55b68586ef8b45439a7dd8dcd2efcd79 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:21.599093 464 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 1381e26725174b9c877075cf5444f566 (table default.Table_b624816662024c55a5f5ef5830b20b7c [id=d2ea84ea59ad4b03ad16bb3052a441f4]). Expected version 2 got 0
I20250114 02:54:21.599938 464 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet f464eabfbc0945fa8a5583f00c1dab40 (table default.Table_4fd891b63924429085d78f78cb177f0e [id=d36c71e7562b4ff1bc8be5cf2955000e]). Expected version 1 got 0
I20250114 02:54:21.601018 464 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 6448eea21ab2422c97da2494658ded3f (table default.Table_0931c459ae794d2091fcc0f51d031548 [id=e279c31bd63b40b9ade03a3b928bf153]). Expected version 2 got 0
I20250114 02:54:21.604547 469 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet 9812754c610548d1b43084bbc94fc3e4
I20250114 02:54:21.605835 469 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:33540 (request call id 200) took 2310 ms. Trace:
I20250114 02:54:21.606343 941 tablet_service.cc:1514] Processing DeleteTablet for tablet 9812754c610548d1b43084bbc94fc3e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:19 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:21.614138 1205 ts_tablet_manager.cc:1428] T 55b68586ef8b45439a7dd8dcd2efcd79 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.056s user 0.006s sys 0.001s
I20250114 02:54:21.614007 1083 tablet_service.cc:1514] Processing DeleteTablet for tablet b63dd90f60a04465b948171e74888e79 with delete_type TABLET_DATA_DELETED (replaced by tablet 59a46bc865cc48df8e98598764fd1b54) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:21.614688 1082 tablet_service.cc:1514] Processing DeleteTablet for tablet 96b06fe4173343dabd726d253b9f9b5f with delete_type TABLET_DATA_DELETED (replaced by tablet e15e6e52d7294aeeb4463b3a76022ec3) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:21.614928 1086 tablet_service.cc:1514] Processing DeleteTablet for tablet 99b8acb1fe594bf3825c7b9121018db6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:16 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:21.615288 1080 tablet_service.cc:1514] Processing DeleteTablet for tablet 00080bea6c8e46e584ccba6dcb0e221b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:17 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:21.615689 1083 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ecf782bb3d54528aa996abca1ea3a15 with delete_type TABLET_DATA_DELETED (replaced by tablet 66bfa92db9874750b3c340e31a06c412) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:21.615895 1085 tablet_service.cc:1514] Processing DeleteTablet for tablet 290ad07ab8be4791a39194909e000843 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:13 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:21.606127 469 rpcz_store.cc:276] 0114 02:54:19.295681 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:19.545297 (+249616us) service_pool.cc:225] Handling call
0114 02:54:19.546470 (+ 1173us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0114 02:54:19.546939 (+ 469us) catalog_manager.cc:2753] Modifying in-memory table state
0114 02:54:19.547043 (+ 104us) catalog_manager.cc:2762] Locking tablets
0114 02:54:19.724850 (+177807us) catalog_manager.cc:2776] Removing table and tablets from system table
0114 02:54:19.996659 (+271809us) catalog_manager.cc:2794] Removing table from by-name map
0114 02:54:19.996899 (+ 240us) catalog_manager.cc:2810] Committing in-memory state
0114 02:54:21.604461 (+1607562us) catalog_manager.cc:2814] Aborting table tasks
0114 02:54:21.605781 (+ 1320us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:19.912414 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:19.912736 (+ 322us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:19.912756 (+ 20us) write_op.cc:435] Acquired schema lock
0114 02:54:19.912768 (+ 12us) tablet.cc:659] Decoding operations
0114 02:54:19.913044 (+ 276us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:19.913081 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:19.913103 (+ 22us) tablet.cc:682] Acquiring locks for 2 operations
0114 02:54:19.913334 (+ 231us) tablet.cc:698] Row locks acquired
0114 02:54:19.913348 (+ 14us) write_op.cc:260] PREPARE: finished
0114 02:54:19.913463 (+ 115us) write_op.cc:270] Start()
0114 02:54:19.913537 (+ 74us) write_op.cc:276] Timestamp: P: 1736823259913390 usec, L: 0
0114 02:54:19.913553 (+ 16us) op_driver.cc:348] REPLICATION: starting
0114 02:54:19.913983 (+ 430us) log.cc:844] Serialized 762 byte log entry
0114 02:54:19.995134 (+ 81151us) op_driver.cc:464] REPLICATION: finished
0114 02:54:19.995375 (+ 241us) write_op.cc:301] APPLY: starting
0114 02:54:19.995431 (+ 56us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:19.995527 (+ 96us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:19.995606 (+ 79us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:19.995840 (+ 234us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:19.995971 (+ 131us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0114 02:54:19.996008 (+ 37us) write_op.cc:312] APPLY: finished
0114 02:54:19.996204 (+ 196us) log.cc:844] Serialized 29 byte log entry
0114 02:54:19.996362 (+ 158us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:19.996425 (+ 63us) write_op.cc:454] Released schema lock
0114 02:54:19.996455 (+ 30us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":100,"apply.run_cpu_time_us":1245,"apply.run_wall_time_us":1246,"num_ops":2,"prepare.queue_time_us":185534,"prepare.run_cpu_time_us":1839,"prepare.run_wall_time_us":74374,"raft.queue_time_us":71816,"raft.run_cpu_time_us":627,"raft.run_wall_time_us":627,"replication_time_us":81500}]]}
I20250114 02:54:21.619086 1207 tablet_bootstrap.cc:654] T bc46815b4803418c95ea9c039a4fae88 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:21.619813 1085 tablet_service.cc:1514] Processing DeleteTablet for tablet 961b6a25a13a4fa2a732a6383dcd8c84 with delete_type TABLET_DATA_DELETED (replaced by tablet 99b940d872ce4781907517c40fe08781) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:21.627293 1205 tablet_bootstrap.cc:492] T 6448eea21ab2422c97da2494658ded3f P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
W20250114 02:54:21.625999 471 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.67:34825 (request call id 41) took 2413 ms (client timeout 1000 ms). Trace:
W20250114 02:54:21.628484 471 rpcz_store.cc:269] 0114 02:54:19.212479 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:19.216851 (+ 4372us) service_pool.cc:225] Handling call
0114 02:54:21.625938 (+2409087us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:20.737843 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:20.738316 (+ 473us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:20.738359 (+ 43us) write_op.cc:435] Acquired schema lock
0114 02:54:20.738380 (+ 21us) tablet.cc:659] Decoding operations
0114 02:54:20.738986 (+ 606us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:20.739033 (+ 47us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:20.739067 (+ 34us) tablet.cc:682] Acquiring locks for 1 operations
0114 02:54:20.739237 (+ 170us) tablet.cc:698] Row locks acquired
0114 02:54:20.739255 (+ 18us) write_op.cc:260] PREPARE: finished
0114 02:54:20.739395 (+ 140us) write_op.cc:270] Start()
0114 02:54:20.739515 (+ 120us) write_op.cc:276] Timestamp: P: 1736823260739346 usec, L: 0
0114 02:54:20.739535 (+ 20us) op_driver.cc:348] REPLICATION: starting
0114 02:54:20.740144 (+ 609us) log.cc:844] Serialized 530 byte log entry
0114 02:54:20.833121 (+ 92977us) op_driver.cc:464] REPLICATION: finished
0114 02:54:20.833331 (+ 210us) write_op.cc:301] APPLY: starting
0114 02:54:20.833376 (+ 45us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:20.833458 (+ 82us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:20.833537 (+ 79us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:20.833670 (+ 133us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:20.833780 (+ 110us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:20.833819 (+ 39us) write_op.cc:312] APPLY: finished
0114 02:54:20.833971 (+ 152us) log.cc:844] Serialized 23 byte log entry
0114 02:54:20.834128 (+ 157us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:20.834188 (+ 60us) write_op.cc:454] Released schema lock
0114 02:54:20.834214 (+ 26us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":23,"child_traces":[["op",{"apply.queue_time_us":77,"apply.run_cpu_time_us":1124,"apply.run_wall_time_us":16852,"num_ops":1,"prepare.queue_time_us":190297,"prepare.run_cpu_time_us":2341,"prepare.run_wall_time_us":3970,"raft.queue_time_us":90,"raft.run_cpu_time_us":13,"raft.run_wall_time_us":14,"replication_time_us":93487}]]}
I20250114 02:54:21.615731 1084 tablet_service.cc:1514] Processing DeleteTablet for tablet 9812754c610548d1b43084bbc94fc3e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:19 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:21.637565 463 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_1142a1a45945449ea16f5501c3adeb8b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:21.647796 463 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1142a1a45945449ea16f5501c3adeb8b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:21.656867 1205 tablet_bootstrap.cc:654] T 6448eea21ab2422c97da2494658ded3f P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:21.659482 1207 tablet_bootstrap.cc:492] T bc46815b4803418c95ea9c039a4fae88 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:21.659904 1207 ts_tablet_manager.cc:1397] T bc46815b4803418c95ea9c039a4fae88 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.110s user 0.010s sys 0.000s
I20250114 02:54:21.662118 1207 raft_consensus.cc:357] T bc46815b4803418c95ea9c039a4fae88 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:21.662739 1207 raft_consensus.cc:383] T bc46815b4803418c95ea9c039a4fae88 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:21.662978 1207 raft_consensus.cc:738] T bc46815b4803418c95ea9c039a4fae88 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:21.665163 471 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_529c38ed585343acb0f42ac60f171b59"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:21.666040 1595 ts_tablet_manager.cc:1918] T a456420f78714cf895528e19dfe0ee8d P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:21.666381 1595 log.cc:1198] T a456420f78714cf895528e19dfe0ee8d P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/a456420f78714cf895528e19dfe0ee8d
W20250114 02:54:21.666640 471 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_529c38ed585343acb0f42ac60f171b59 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:21.667079 1595 ts_tablet_manager.cc:1939] T a456420f78714cf895528e19dfe0ee8d P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
I20250114 02:54:21.669847 1702 ts_tablet_manager.cc:1918] T a456420f78714cf895528e19dfe0ee8d P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:21.670581 1702 log.cc:1198] T a456420f78714cf895528e19dfe0ee8d P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/a456420f78714cf895528e19dfe0ee8d
I20250114 02:54:21.671340 1702 ts_tablet_manager.cc:1939] T a456420f78714cf895528e19dfe0ee8d P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:21.673847 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet a456420f78714cf895528e19dfe0ee8d (table default.Table_fd43493446ca49a48e8a970bb92ea239 [id=6dbd2db73b5443a2af5d3b5056ab0169]) successfully deleted
I20250114 02:54:21.678352 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet a456420f78714cf895528e19dfe0ee8d (table default.Table_fd43493446ca49a48e8a970bb92ea239 [id=6dbd2db73b5443a2af5d3b5056ab0169]) successfully deleted
I20250114 02:54:21.679951 463 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_fee886c2440a47bfa5a5875d58ef45ad" } new_table_name: "default.Table_1c9b7896941e4289b1c12e20c210c2c1" modify_external_catalogs: true
I20250114 02:54:21.675985 1207 consensus_queue.cc:260] T bc46815b4803418c95ea9c039a4fae88 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:21.681973 1595 tablet_replica.cc:331] T ec3285525faf482d8739f1bedd653503 P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:21.682713 1595 raft_consensus.cc:2238] T ec3285525faf482d8739f1bedd653503 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Raft consensus shutting down.
I20250114 02:54:21.683552 1595 raft_consensus.cc:2267] T ec3285525faf482d8739f1bedd653503 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:21.685032 1784 raft_consensus.cc:491] T bc46815b4803418c95ea9c039a4fae88 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:21.685431 1784 raft_consensus.cc:513] T bc46815b4803418c95ea9c039a4fae88 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:21.688762 1207 ts_tablet_manager.cc:1428] T bc46815b4803418c95ea9c039a4fae88 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.029s user 0.006s sys 0.000s
I20250114 02:54:21.692013 1205 tablet_bootstrap.cc:492] T 6448eea21ab2422c97da2494658ded3f P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:21.695327 1207 tablet_bootstrap.cc:492] T 6448eea21ab2422c97da2494658ded3f P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:21.709736 1828 raft_consensus.cc:491] T 6448eea21ab2422c97da2494658ded3f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:21.709345 788 tablet_service.cc:1467] Processing CreateTablet for tablet 0b8989d340fa4ae992ef97e226d41110 (DEFAULT_TABLE table=default.Table_24cfc302131c4035b4c8e0fdc03625fa [id=d772a301194a4e9ab90cabfc05104081]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:21.711473 788 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b8989d340fa4ae992ef97e226d41110. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:21.711788 789 tablet_service.cc:1467] Processing CreateTablet for tablet d2a7b2bee5c1440fb705b172c18a9247 (DEFAULT_TABLE table=default.Table_910b0ec1594a4dfcbb342055cab3eb65 [id=4d2f7fabf8a44155a586a2b79a8fca53]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:21.713312 789 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2a7b2bee5c1440fb705b172c18a9247. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:21.719326 959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc46815b4803418c95ea9c039a4fae88" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:21.720126 959 raft_consensus.cc:2463] T bc46815b4803418c95ea9c039a4fae88 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 0.
I20250114 02:54:21.721479 796 tablet_service.cc:1514] Processing DeleteTablet for tablet 9812754c610548d1b43084bbc94fc3e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:19 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:21.722747 790 tablet_service.cc:1514] Processing DeleteTablet for tablet ec3285525faf482d8739f1bedd653503 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:20 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:21.723575 1828 raft_consensus.cc:513] T 6448eea21ab2422c97da2494658ded3f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:21.725370 1205 ts_tablet_manager.cc:1397] T 6448eea21ab2422c97da2494658ded3f P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.099s user 0.009s sys 0.000s
I20250114 02:54:21.725783 1040 leader_election.cc:304] T bc46815b4803418c95ea9c039a4fae88 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:21.726622 1819 raft_consensus.cc:2798] T bc46815b4803418c95ea9c039a4fae88 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:21.726588 959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6448eea21ab2422c97da2494658ded3f" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:21.726927 1819 raft_consensus.cc:491] T bc46815b4803418c95ea9c039a4fae88 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:21.727206 1819 raft_consensus.cc:3054] T bc46815b4803418c95ea9c039a4fae88 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
W20250114 02:54:21.727887 751 leader_election.cc:343] T 6448eea21ab2422c97da2494658ded3f P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:21.731606 1594 ts_tablet_manager.cc:1939] T 290ad07ab8be4791a39194909e000843 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:21.733117 1594 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 143) took 2587 ms. Trace:
I20250114 02:54:21.733564 1594 rpcz_store.cc:276] 0114 02:54:19.145335 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:19.145425 (+ 90us) service_pool.cc:225] Handling call
0114 02:54:21.178537 (+2033112us) ts_tablet_manager.cc:1145] Deleting tablet 290ad07ab8be4791a39194909e000843
0114 02:54:21.178574 (+ 37us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:21.501697 (+323123us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:21.543230 (+ 41533us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:21.732975 (+189745us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":107530,"tablet-delete.queue_time_us":2032630}
I20250114 02:54:21.738548 1206 tablet_bootstrap.cc:492] T 0304df55431e42e1a999ed17dc35e6c0 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:21.739210 1206 ts_tablet_manager.cc:1397] T 0304df55431e42e1a999ed17dc35e6c0 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.324s user 0.005s sys 0.004s
I20250114 02:54:21.735473 1594 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 141) took 2590 ms. Trace:
I20250114 02:54:21.742247 1786 raft_consensus.cc:491] T 55b68586ef8b45439a7dd8dcd2efcd79 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:21.742496 1595 ts_tablet_manager.cc:1905] T ec3285525faf482d8739f1bedd653503 P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:21.742771 1786 raft_consensus.cc:513] T 55b68586ef8b45439a7dd8dcd2efcd79 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:21.744974 1786 leader_election.cc:290] T 55b68586ef8b45439a7dd8dcd2efcd79 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:21.749310 1702 tablet_replica.cc:331] T ec3285525faf482d8739f1bedd653503 P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:21.749799 1101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55b68586ef8b45439a7dd8dcd2efcd79" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:21.750761 1101 raft_consensus.cc:2388] T 55b68586ef8b45439a7dd8dcd2efcd79 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 525198593c8e47afaee54e74267b87e3 in current term 1: Already voted for candidate 90d6c2c58bb54e0eafb365625ac33871 in this term.
I20250114 02:54:21.752926 1820 consensus_queue.cc:1035] T 66bfa92db9874750b3c340e31a06c412 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:54:21.754341 1797 consensus_queue.cc:1035] T 26328f8e32bc49ed9b224f0e08b9f949 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250114 02:54:21.754946 455 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.124:39105, user_credentials={real_user=slave}} blocked reactor thread for 51211.4us
W20250114 02:54:21.739804 751 consensus_peers.cc:487] T a456420f78714cf895528e19dfe0ee8d P 819f9e3412e54d44abb11d8b3ca877c2 -> Peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Couldn't send request to peer 525198593c8e47afaee54e74267b87e3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a456420f78714cf895528e19dfe0ee8d. This is attempt 1: this message will repeat every 5th retry.
I20250114 02:54:21.760208 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55b68586ef8b45439a7dd8dcd2efcd79" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:21.761133 818 raft_consensus.cc:3054] T 55b68586ef8b45439a7dd8dcd2efcd79 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:21.727736 1205 raft_consensus.cc:357] T 6448eea21ab2422c97da2494658ded3f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:21.762437 1205 raft_consensus.cc:383] T 6448eea21ab2422c97da2494658ded3f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:21.762709 1205 raft_consensus.cc:738] T 6448eea21ab2422c97da2494658ded3f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:21.765861 1207 tablet_bootstrap.cc:654] T 6448eea21ab2422c97da2494658ded3f P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:21.740154 1594 rpcz_store.cc:276] 0114 02:54:19.144861 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:19.145448 (+ 587us) service_pool.cc:225] Handling call
0114 02:54:21.735024 (+2589576us) ts_tablet_manager.cc:1145] Deleting tablet 9589afd0094545c1801b7a0b5eb8698f
0114 02:54:21.735148 (+ 124us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:21.735422 (+ 274us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2588337}
I20250114 02:54:21.770630 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet 290ad07ab8be4791a39194909e000843 (table default.Table_02b3fc3bfe484e3a9c5d5e9114deb368 [id=29d8f8e2417d403aa619839418c94207]) successfully deleted
W20250114 02:54:21.771488 898 leader_election.cc:343] T bc46815b4803418c95ea9c039a4fae88 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Not found: Tablet not found: bc46815b4803418c95ea9c039a4fae88
I20250114 02:54:21.771916 898 leader_election.cc:304] T bc46815b4803418c95ea9c039a4fae88 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 525198593c8e47afaee54e74267b87e3; no voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:21.773931 1786 raft_consensus.cc:2743] T bc46815b4803418c95ea9c039a4fae88 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:21.775331 1099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6448eea21ab2422c97da2494658ded3f" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:21.776304 1828 leader_election.cc:290] T 6448eea21ab2422c97da2494658ded3f P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:21.777339 815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55b68586ef8b45439a7dd8dcd2efcd79" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
W20250114 02:54:21.781297 1042 leader_election.cc:343] T bc46815b4803418c95ea9c039a4fae88 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Not found: Tablet not found: bc46815b4803418c95ea9c039a4fae88
I20250114 02:54:21.784328 1784 leader_election.cc:290] T bc46815b4803418c95ea9c039a4fae88 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:21.785256 1205 consensus_queue.cc:260] T 6448eea21ab2422c97da2494658ded3f P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:21.786836 1205 ts_tablet_manager.cc:1428] T 6448eea21ab2422c97da2494658ded3f P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.061s user 0.004s sys 0.000s
I20250114 02:54:21.787125 1594 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 145) took 2639 ms. Trace:
I20250114 02:54:21.787578 898 leader_election.cc:304] T 55b68586ef8b45439a7dd8dcd2efcd79 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 525198593c8e47afaee54e74267b87e3; no voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:21.787464 1594 rpcz_store.cc:276] 0114 02:54:19.147598 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:19.147684 (+ 86us) service_pool.cc:225] Handling call
0114 02:54:21.771729 (+2624045us) ts_tablet_manager.cc:1145] Deleting tablet 276942e5e54246c79f0d4f08297f58ea
0114 02:54:21.786857 (+ 15128us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:21.787071 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2621186}
I20250114 02:54:21.773567 1206 raft_consensus.cc:357] T 0304df55431e42e1a999ed17dc35e6c0 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:21.789633 1206 raft_consensus.cc:383] T 0304df55431e42e1a999ed17dc35e6c0 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:21.789878 1206 raft_consensus.cc:738] T 0304df55431e42e1a999ed17dc35e6c0 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:21.789458 1786 raft_consensus.cc:3054] T 55b68586ef8b45439a7dd8dcd2efcd79 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
W20250114 02:54:21.785743 753 leader_election.cc:343] T 6448eea21ab2422c97da2494658ded3f P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:21.790418 1206 consensus_queue.cc:260] T 0304df55431e42e1a999ed17dc35e6c0 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:21.790731 753 leader_election.cc:304] T 6448eea21ab2422c97da2494658ded3f P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:21.791638 1741 raft_consensus.cc:2743] T 6448eea21ab2422c97da2494658ded3f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250114 02:54:21.792992 456 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet edc69b66bea54625a4b15867b75986b7 (table default.Table_0a0bda55a4a9476c98d961822b8a19bd [id=fb2be5c1ca9444a197f3315ea9c59b09], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet edc69b66bea54625a4b15867b75986b7 consensus state has no leader
I20250114 02:54:21.793918 1819 raft_consensus.cc:513] T bc46815b4803418c95ea9c039a4fae88 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:21.796753 959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc46815b4803418c95ea9c039a4fae88" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
I20250114 02:54:21.797464 959 raft_consensus.cc:3054] T bc46815b4803418c95ea9c039a4fae88 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:21.803970 1786 raft_consensus.cc:2743] T 55b68586ef8b45439a7dd8dcd2efcd79 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:21.804665 1594 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 144) took 2657 ms. Trace:
I20250114 02:54:21.805934 1594 rpcz_store.cc:276] 0114 02:54:19.147357 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:19.148262 (+ 905us) service_pool.cc:225] Handling call
0114 02:54:21.788409 (+2640147us) ts_tablet_manager.cc:1145] Deleting tablet b63dd90f60a04465b948171e74888e79
0114 02:54:21.804393 (+ 15984us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:21.804616 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2639714}
W20250114 02:54:21.808815 1042 leader_election.cc:343] T bc46815b4803418c95ea9c039a4fae88 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Not found: Tablet not found: bc46815b4803418c95ea9c039a4fae88
I20250114 02:54:21.811954 1819 leader_election.cc:290] T bc46815b4803418c95ea9c039a4fae88 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Requested vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:21.804330 959 raft_consensus.cc:2463] T bc46815b4803418c95ea9c039a4fae88 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1.
I20250114 02:54:21.810175 1205 tablet_bootstrap.cc:492] T 0f1ea1608213492c9a216757896ff2a4 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:21.813611 1040 leader_election.cc:304] T bc46815b4803418c95ea9c039a4fae88 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters: 819f9e3412e54d44abb11d8b3ca877c2
I20250114 02:54:21.813747 1702 raft_consensus.cc:2238] T ec3285525faf482d8739f1bedd653503 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:21.814270 1702 raft_consensus.cc:2267] T ec3285525faf482d8739f1bedd653503 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250114 02:54:21.817080 458 catalog_manager.cc:4839] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet 9589afd0094545c1801b7a0b5eb8698f because the tablet was not found. No further retry: Not found: Tablet not found: 9589afd0094545c1801b7a0b5eb8698f
W20250114 02:54:21.818874 458 catalog_manager.cc:4839] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet 276942e5e54246c79f0d4f08297f58ea because the tablet was not found. No further retry: Not found: Tablet not found: 276942e5e54246c79f0d4f08297f58ea
W20250114 02:54:21.819737 458 catalog_manager.cc:4839] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet b63dd90f60a04465b948171e74888e79 because the tablet was not found. No further retry: Not found: Tablet not found: b63dd90f60a04465b948171e74888e79
I20250114 02:54:21.820402 470 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet 6e858a8449444ef999b9bbef612249d3
I20250114 02:54:21.825402 1207 tablet_bootstrap.cc:492] T 6448eea21ab2422c97da2494658ded3f P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:21.825799 1207 ts_tablet_manager.cc:1397] T 6448eea21ab2422c97da2494658ded3f P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.131s user 0.009s sys 0.000s
I20250114 02:54:21.826205 1819 raft_consensus.cc:2798] T bc46815b4803418c95ea9c039a4fae88 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:21.792150 1206 ts_tablet_manager.cc:1428] T 0304df55431e42e1a999ed17dc35e6c0 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.053s user 0.006s sys 0.001s
W20250114 02:54:21.832536 463 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a52d6749ff834b62a8e20a2f45935240 consensus state has no leader
I20250114 02:54:21.834815 933 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e858a8449444ef999b9bbef612249d3 with delete_type TABLET_DATA_DELETED (replaced by tablet 797d6cdf36704c98aa106aac87a945ca) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:21.835772 1081 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e858a8449444ef999b9bbef612249d3 with delete_type TABLET_DATA_DELETED (replaced by tablet 797d6cdf36704c98aa106aac87a945ca) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:21.816121 1702 ts_tablet_manager.cc:1905] T ec3285525faf482d8739f1bedd653503 P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:21.821972 1205 tablet_bootstrap.cc:654] T 0f1ea1608213492c9a216757896ff2a4 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:21.845899 463 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_8c98f977242949aa90be45b2f66d5679"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:21.847370 463 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8c98f977242949aa90be45b2f66d5679 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:21.828087 1207 raft_consensus.cc:357] T 6448eea21ab2422c97da2494658ded3f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:21.843892 1819 raft_consensus.cc:695] T bc46815b4803418c95ea9c039a4fae88 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Becoming Leader. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Running, Role: LEADER
I20250114 02:54:21.849083 1819 consensus_queue.cc:237] T bc46815b4803418c95ea9c039a4fae88 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:21.851593 470 catalog_manager.cc:6596] ReplaceTablet: tablet 6e858a8449444ef999b9bbef612249d3 deleted and replaced by tablet 797d6cdf36704c98aa106aac87a945ca
I20250114 02:54:21.848539 1207 raft_consensus.cc:383] T 6448eea21ab2422c97da2494658ded3f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:21.853406 1207 raft_consensus.cc:738] T 6448eea21ab2422c97da2494658ded3f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:21.854580 1207 consensus_queue.cc:260] T 6448eea21ab2422c97da2494658ded3f P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:21.856482 1594 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 146) took 2708 ms. Trace:
I20250114 02:54:21.858049 1207 ts_tablet_manager.cc:1428] T 6448eea21ab2422c97da2494658ded3f P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.032s user 0.007s sys 0.000s
I20250114 02:54:21.858800 1206 tablet_bootstrap.cc:492] T 71bd00f7d8ac41839f36fd73997631ad P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:21.856804 1594 rpcz_store.cc:276] 0114 02:54:19.147838 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:19.147973 (+ 135us) service_pool.cc:225] Handling call
0114 02:54:21.806932 (+2658959us) ts_tablet_manager.cc:1145] Deleting tablet 96b06fe4173343dabd726d253b9f9b5f
0114 02:54:21.831155 (+ 24223us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:21.856426 (+ 25271us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2657656}
I20250114 02:54:21.864446 1205 tablet_bootstrap.cc:492] T 0f1ea1608213492c9a216757896ff2a4 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:21.869330 790 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e858a8449444ef999b9bbef612249d3 with delete_type TABLET_DATA_DELETED (replaced by tablet 797d6cdf36704c98aa106aac87a945ca) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:21.880460 470 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_6a4fe39298514187b65def5841d3a992" } modify_external_catalogs: true
I20250114 02:54:21.881047 470 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_6a4fe39298514187b65def5841d3a992" } modify_external_catalogs: true
I20250114 02:54:21.883318 1205 ts_tablet_manager.cc:1397] T 0f1ea1608213492c9a216757896ff2a4 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.073s user 0.009s sys 0.002s
I20250114 02:54:21.884078 959 raft_consensus.cc:3054] T 26328f8e32bc49ed9b224f0e08b9f949 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:21.885759 1205 raft_consensus.cc:357] T 0f1ea1608213492c9a216757896ff2a4 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:21.886458 1205 raft_consensus.cc:383] T 0f1ea1608213492c9a216757896ff2a4 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:21.886996 1205 raft_consensus.cc:738] T 0f1ea1608213492c9a216757896ff2a4 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:21.887089 1594 tablet_replica.cc:331] T ba215eb8422d4f879dd0bf5996543962 P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:21.886714 1595 ts_tablet_manager.cc:1918] T ec3285525faf482d8739f1bedd653503 P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:21.887696 1595 log.cc:1198] T ec3285525faf482d8739f1bedd653503 P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/ec3285525faf482d8739f1bedd653503
I20250114 02:54:21.888537 1595 ts_tablet_manager.cc:1939] T ec3285525faf482d8739f1bedd653503 P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
I20250114 02:54:21.863456 818 raft_consensus.cc:2463] T 55b68586ef8b45439a7dd8dcd2efcd79 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1.
I20250114 02:54:21.890945 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet ec3285525faf482d8739f1bedd653503 (table default.Table_4598efdd6ff348d081ed8c3e7dc3130f [id=fdd50384f4894e20b21f109c6a33864c]) successfully deleted
W20250114 02:54:21.892379 458 catalog_manager.cc:4839] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet b63dd90f60a04465b948171e74888e79 because the tablet was not found. No further retry: Not found: Tablet not found: b63dd90f60a04465b948171e74888e79
W20250114 02:54:21.892992 458 catalog_manager.cc:4839] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 96b06fe4173343dabd726d253b9f9b5f because the tablet was not found. No further retry: Not found: Tablet not found: 96b06fe4173343dabd726d253b9f9b5f
W20250114 02:54:21.893518 458 catalog_manager.cc:4839] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 99b8acb1fe594bf3825c7b9121018db6 because the tablet was not found. No further retry: Not found: Tablet not found: 99b8acb1fe594bf3825c7b9121018db6
W20250114 02:54:21.894692 458 catalog_manager.cc:4839] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 00080bea6c8e46e584ccba6dcb0e221b because the tablet was not found. No further retry: Not found: Tablet not found: 00080bea6c8e46e584ccba6dcb0e221b
I20250114 02:54:21.893560 1594 raft_consensus.cc:2238] T ba215eb8422d4f879dd0bf5996543962 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 LEADER]: Raft consensus shutting down.
W20250114 02:54:21.895521 458 catalog_manager.cc:4839] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 7ecf782bb3d54528aa996abca1ea3a15 because the tablet was not found. No further retry: Not found: Tablet not found: 7ecf782bb3d54528aa996abca1ea3a15
I20250114 02:54:21.897038 1841 consensus_queue.cc:1035] T 26328f8e32bc49ed9b224f0e08b9f949 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:21.898476 1207 tablet_bootstrap.cc:492] T 0f1ea1608213492c9a216757896ff2a4 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
W20250114 02:54:21.898370 751 consensus_peers.cc:487] T ba215eb8422d4f879dd0bf5996543962 P 819f9e3412e54d44abb11d8b3ca877c2 -> Peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Couldn't send request to peer 525198593c8e47afaee54e74267b87e3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ba215eb8422d4f879dd0bf5996543962. This is attempt 1: this message will repeat every 5th retry.
I20250114 02:54:21.901639 1042 leader_election.cc:304] T 55b68586ef8b45439a7dd8dcd2efcd79 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters: 525198593c8e47afaee54e74267b87e3
I20250114 02:54:21.902279 1841 raft_consensus.cc:2798] T 55b68586ef8b45439a7dd8dcd2efcd79 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:21.902729 1841 raft_consensus.cc:695] T 55b68586ef8b45439a7dd8dcd2efcd79 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Becoming Leader. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Running, Role: LEADER
I20250114 02:54:21.903628 1841 consensus_queue.cc:237] T 55b68586ef8b45439a7dd8dcd2efcd79 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
W20250114 02:54:21.908291 458 catalog_manager.cc:4839] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet 96b06fe4173343dabd726d253b9f9b5f because the tablet was not found. No further retry: Not found: Tablet not found: 96b06fe4173343dabd726d253b9f9b5f
I20250114 02:54:21.914249 1206 tablet_bootstrap.cc:654] T 71bd00f7d8ac41839f36fd73997631ad P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:21.948522 471 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_1a9735a294c2436e84ec9db5e0aa44e4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:21.950299 471 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1a9735a294c2436e84ec9db5e0aa44e4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:21.951803 1205 consensus_queue.cc:260] T 0f1ea1608213492c9a216757896ff2a4 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:21.962581 1594 raft_consensus.cc:2267] T ba215eb8422d4f879dd0bf5996543962 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250114 02:54:21.966447 458 catalog_manager.cc:4839] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 290ad07ab8be4791a39194909e000843 because the tablet was not found. No further retry: Not found: Tablet not found: 290ad07ab8be4791a39194909e000843
W20250114 02:54:21.967828 458 catalog_manager.cc:4839] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 961b6a25a13a4fa2a732a6383dcd8c84 because the tablet was not found. No further retry: Not found: Tablet not found: 961b6a25a13a4fa2a732a6383dcd8c84
I20250114 02:54:21.969717 1207 tablet_bootstrap.cc:654] T 0f1ea1608213492c9a216757896ff2a4 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:21.982100 1849 tablet.cc:1721] T 26328f8e32bc49ed9b224f0e08b9f949 P 525198593c8e47afaee54e74267b87e3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250114 02:54:21.983464 1850 tablet.cc:1721] T 26328f8e32bc49ed9b224f0e08b9f949 P 90d6c2c58bb54e0eafb365625ac33871: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250114 02:54:21.984468 1786 raft_consensus.cc:491] T 66bfa92db9874750b3c340e31a06c412 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:21.985137 1786 raft_consensus.cc:513] T 66bfa92db9874750b3c340e31a06c412 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:21.985817 1205 ts_tablet_manager.cc:1428] T 0f1ea1608213492c9a216757896ff2a4 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.102s user 0.006s sys 0.000s
I20250114 02:54:21.986773 1786 leader_election.cc:290] T 66bfa92db9874750b3c340e31a06c412 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:21.988633 1099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66bfa92db9874750b3c340e31a06c412" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:21.993381 463 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_f851d0fbfd4e4095912dbe3a96fd7142" } modify_external_catalogs: true
I20250114 02:54:21.993865 463 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_f851d0fbfd4e4095912dbe3a96fd7142" } modify_external_catalogs: true
W20250114 02:54:21.964520 753 consensus_peers.cc:487] T a456420f78714cf895528e19dfe0ee8d P 819f9e3412e54d44abb11d8b3ca877c2 -> Peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Couldn't send request to peer 90d6c2c58bb54e0eafb365625ac33871. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a456420f78714cf895528e19dfe0ee8d. This is attempt 1: this message will repeat every 5th retry.
I20250114 02:54:21.999037 961 raft_consensus.cc:3054] T 66bfa92db9874750b3c340e31a06c412 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:22.000188 1594 ts_tablet_manager.cc:1905] T ba215eb8422d4f879dd0bf5996543962 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:21.984050 1595 tablet_replica.cc:331] T 9812754c610548d1b43084bbc94fc3e4 P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:22.014186 1595 raft_consensus.cc:2238] T 9812754c610548d1b43084bbc94fc3e4 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:22.018354 1595 raft_consensus.cc:2267] T 9812754c610548d1b43084bbc94fc3e4 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:22.018450 1205 tablet_bootstrap.cc:492] T 71bd00f7d8ac41839f36fd73997631ad P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:22.021073 1595 ts_tablet_manager.cc:1905] T 9812754c610548d1b43084bbc94fc3e4 P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:22.022758 815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66bfa92db9874750b3c340e31a06c412" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:22.024339 898 leader_election.cc:304] T 66bfa92db9874750b3c340e31a06c412 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 525198593c8e47afaee54e74267b87e3; no voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:22.051241 1702 ts_tablet_manager.cc:1918] T ec3285525faf482d8739f1bedd653503 P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:22.051595 1702 log.cc:1198] T ec3285525faf482d8739f1bedd653503 P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/ec3285525faf482d8739f1bedd653503
I20250114 02:54:22.052291 1702 ts_tablet_manager.cc:1939] T ec3285525faf482d8739f1bedd653503 P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:22.054423 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet ec3285525faf482d8739f1bedd653503 (table default.Table_4598efdd6ff348d081ed8c3e7dc3130f [id=fdd50384f4894e20b21f109c6a33864c]) successfully deleted
I20250114 02:54:22.055764 1206 tablet_bootstrap.cc:492] T 71bd00f7d8ac41839f36fd73997631ad P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:22.056156 1206 ts_tablet_manager.cc:1397] T 71bd00f7d8ac41839f36fd73997631ad P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.198s user 0.005s sys 0.007s
I20250114 02:54:22.068495 1206 raft_consensus.cc:357] T 71bd00f7d8ac41839f36fd73997631ad P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:22.069156 1206 raft_consensus.cc:383] T 71bd00f7d8ac41839f36fd73997631ad P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:22.069387 1206 raft_consensus.cc:738] T 71bd00f7d8ac41839f36fd73997631ad P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:22.074786 471 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_df476a4b13be4c11a896663f6211624d" } new_table_name: "default.Table_2befe573a994473abe0556ba7e713686" modify_external_catalogs: true
I20250114 02:54:22.086936 1850 tablet.cc:1721] T aad96f2507974023aaa988e6950feb7a P 90d6c2c58bb54e0eafb365625ac33871: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250114 02:54:22.089915 1786 raft_consensus.cc:2743] T 66bfa92db9874750b3c340e31a06c412 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:22.092293 1857 tablet.cc:1721] T aad96f2507974023aaa988e6950feb7a P 819f9e3412e54d44abb11d8b3ca877c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250114 02:54:22.094026 1784 consensus_queue.cc:1035] T 66bfa92db9874750b3c340e31a06c412 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:22.076107 1855 tablet.cc:1721] T 26328f8e32bc49ed9b224f0e08b9f949 P 819f9e3412e54d44abb11d8b3ca877c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250114 02:54:22.096951 1207 tablet_bootstrap.cc:492] T 0f1ea1608213492c9a216757896ff2a4 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:22.097441 1207 ts_tablet_manager.cc:1397] T 0f1ea1608213492c9a216757896ff2a4 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.199s user 0.010s sys 0.000s
I20250114 02:54:22.101840 1207 raft_consensus.cc:357] T 0f1ea1608213492c9a216757896ff2a4 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:22.023743 1205 tablet_bootstrap.cc:654] T 71bd00f7d8ac41839f36fd73997631ad P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:22.111009 1206 consensus_queue.cc:260] T 71bd00f7d8ac41839f36fd73997631ad P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:22.118228 1206 ts_tablet_manager.cc:1428] T 71bd00f7d8ac41839f36fd73997631ad P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.062s user 0.005s sys 0.000s
I20250114 02:54:22.109817 1849 tablet.cc:1721] T aad96f2507974023aaa988e6950feb7a P 525198593c8e47afaee54e74267b87e3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250114 02:54:22.123270 1786 consensus_queue.cc:1035] T 321b9ad6d1ac4d1c80c7a01faf2015af P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:22.136426 1206 tablet_bootstrap.cc:492] T f2b6812e8c1d44529d0284be490df648 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
W20250114 02:54:22.145401 471 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet df045d80d8ce4ff194a6d2039a44bce6 consensus state has no leader
I20250114 02:54:22.104132 1595 ts_tablet_manager.cc:1918] T 9812754c610548d1b43084bbc94fc3e4 P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:22.149564 1595 log.cc:1198] T 9812754c610548d1b43084bbc94fc3e4 P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/9812754c610548d1b43084bbc94fc3e4
I20250114 02:54:22.150504 1595 ts_tablet_manager.cc:1939] T 9812754c610548d1b43084bbc94fc3e4 P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
I20250114 02:54:22.151058 1207 raft_consensus.cc:383] T 0f1ea1608213492c9a216757896ff2a4 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:22.153707 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet 9812754c610548d1b43084bbc94fc3e4 (table default.Table_2217d3b1dcf4455d9103cad045518573 [id=2199e16ab337450ebc7c664318521edf]) successfully deleted
W20250114 02:54:22.154407 458 catalog_manager.cc:4852] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 6e858a8449444ef999b9bbef612249d3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e858a8449444ef999b9bbef612249d3 already in progress: creating tablet
I20250114 02:54:22.152948 1207 raft_consensus.cc:738] T 0f1ea1608213492c9a216757896ff2a4 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:22.156682 1702 tablet_replica.cc:331] T 9812754c610548d1b43084bbc94fc3e4 P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:22.157408 1702 raft_consensus.cc:2238] T 9812754c610548d1b43084bbc94fc3e4 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:22.157851 1702 raft_consensus.cc:2267] T 9812754c610548d1b43084bbc94fc3e4 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:22.158411 1207 consensus_queue.cc:260] T 0f1ea1608213492c9a216757896ff2a4 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:22.159497 1702 ts_tablet_manager.cc:1905] T 9812754c610548d1b43084bbc94fc3e4 P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:22.162703 1206 tablet_bootstrap.cc:654] T f2b6812e8c1d44529d0284be490df648 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:22.156071 471 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_b41debc5d26d4d5aa95793afabcec087"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:22.189958 1783 consensus_queue.cc:1035] T 321b9ad6d1ac4d1c80c7a01faf2015af P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:22.196487 1207 ts_tablet_manager.cc:1428] T 0f1ea1608213492c9a216757896ff2a4 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.099s user 0.008s sys 0.000s
I20250114 02:54:22.198276 1207 tablet_bootstrap.cc:492] T 71bd00f7d8ac41839f36fd73997631ad P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:22.201663 1205 tablet_bootstrap.cc:492] T 71bd00f7d8ac41839f36fd73997631ad P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:22.202251 1205 ts_tablet_manager.cc:1397] T 71bd00f7d8ac41839f36fd73997631ad P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.184s user 0.010s sys 0.000s
I20250114 02:54:22.209220 1205 raft_consensus.cc:357] T 71bd00f7d8ac41839f36fd73997631ad P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
W20250114 02:54:22.211036 471 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b41debc5d26d4d5aa95793afabcec087 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:22.210233 1205 raft_consensus.cc:383] T 71bd00f7d8ac41839f36fd73997631ad P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:22.199568 1594 ts_tablet_manager.cc:1918] T ba215eb8422d4f879dd0bf5996543962 P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:22.228809 1205 raft_consensus.cc:738] T 71bd00f7d8ac41839f36fd73997631ad P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:22.229321 1594 log.cc:1198] T ba215eb8422d4f879dd0bf5996543962 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/ba215eb8422d4f879dd0bf5996543962
W20250114 02:54:22.249114 1042 consensus_peers.cc:487] T 53a8bb487ad64157ad3baecf5d866eca P 90d6c2c58bb54e0eafb365625ac33871 -> Peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Couldn't send request to peer 819f9e3412e54d44abb11d8b3ca877c2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 53a8bb487ad64157ad3baecf5d866eca. This is attempt 1: this message will repeat every 5th retry.
I20250114 02:54:22.252070 471 master_service.cc:946] ReplaceTablet: received request to replace tablet c86562688d5b4fffb71cbdf7be0afc37 from {username='slave'} at 127.0.0.1:33540
I20250114 02:54:22.229729 1205 consensus_queue.cc:260] T 71bd00f7d8ac41839f36fd73997631ad P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:22.262423 1205 ts_tablet_manager.cc:1428] T 71bd00f7d8ac41839f36fd73997631ad P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.060s user 0.005s sys 0.000s
I20250114 02:54:22.264775 1207 tablet_bootstrap.cc:654] T 71bd00f7d8ac41839f36fd73997631ad P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:22.264976 1594 ts_tablet_manager.cc:1939] T ba215eb8422d4f879dd0bf5996543962 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:22.266922 1594 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 147) took 3091 ms. Trace:
I20250114 02:54:22.267213 1594 rpcz_store.cc:276] 0114 02:54:19.175108 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:19.175233 (+ 125us) service_pool.cc:225] Handling call
0114 02:54:21.873360 (+2698127us) ts_tablet_manager.cc:1145] Deleting tablet ba215eb8422d4f879dd0bf5996543962
0114 02:54:21.886936 (+ 13576us) spinlock_profiling.cc:243] Waited 4.36 ms on lock 0x7b5800001898. stack: 00007f66faa37f63 00007f66faa37e60 00007f66fa5ae584 00000000004fcd24 00000000004fccb9 00007f66fa8ad911 00007f66faa2c991 00007f66faa2cfc2 00007f67036bf999 00007f67029245f8 00007f67029e83e9 00007f67029e89e3 00007f6702a18ca0 00007f67029ffe6d 00007f67029ffe19 00007f67029ffda9
0114 02:54:21.886979 (+ 43us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:22.082750 (+195771us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:22.199317 (+116567us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:22.266868 (+ 67551us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10039040,"tablet-delete.queue_time_us":2685941}
I20250114 02:54:22.278028 1206 tablet_bootstrap.cc:492] T f2b6812e8c1d44529d0284be490df648 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:22.278537 1206 ts_tablet_manager.cc:1397] T f2b6812e8c1d44529d0284be490df648 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.160s user 0.007s sys 0.003s
I20250114 02:54:22.280786 1206 raft_consensus.cc:357] T f2b6812e8c1d44529d0284be490df648 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:22.281450 1206 raft_consensus.cc:383] T f2b6812e8c1d44529d0284be490df648 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:22.281724 1206 raft_consensus.cc:738] T f2b6812e8c1d44529d0284be490df648 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:22.282334 1206 consensus_queue.cc:260] T f2b6812e8c1d44529d0284be490df648 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:22.293895 1784 raft_consensus.cc:491] T 266da813b4514329be0ad97d07b7114f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:22.294474 1784 raft_consensus.cc:513] T 266da813b4514329be0ad97d07b7114f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:22.296658 1784 leader_election.cc:290] T 266da813b4514329be0ad97d07b7114f P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:22.297582 1205 tablet_bootstrap.cc:492] T 044832851f804208b0a371eb0857dbf3 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:22.299768 813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "266da813b4514329be0ad97d07b7114f" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:22.300649 813 raft_consensus.cc:2463] T 266da813b4514329be0ad97d07b7114f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 0.
I20250114 02:54:22.301602 1042 leader_election.cc:304] T 266da813b4514329be0ad97d07b7114f P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters:
W20250114 02:54:22.302204 1040 leader_election.cc:343] T 266da813b4514329be0ad97d07b7114f P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Not found: Tablet not found: 266da813b4514329be0ad97d07b7114f
I20250114 02:54:22.302397 1784 raft_consensus.cc:2798] T 266da813b4514329be0ad97d07b7114f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:22.302796 1784 raft_consensus.cc:491] T 266da813b4514329be0ad97d07b7114f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:22.303146 1784 raft_consensus.cc:3054] T 266da813b4514329be0ad97d07b7114f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:22.305863 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet ba215eb8422d4f879dd0bf5996543962 (table default.Table_bacdf22532c84100a69a38cfa0ad07e5 [id=86304f6b48fd421cb953070ae7c9b0a2]) successfully deleted
I20250114 02:54:22.308657 1205 tablet_bootstrap.cc:654] T 044832851f804208b0a371eb0857dbf3 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
W20250114 02:54:22.309026 753 consensus_peers.cc:487] T 9812754c610548d1b43084bbc94fc3e4 P 819f9e3412e54d44abb11d8b3ca877c2 -> Peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Couldn't send request to peer 90d6c2c58bb54e0eafb365625ac33871. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9812754c610548d1b43084bbc94fc3e4. This is attempt 1: this message will repeat every 5th retry.
I20250114 02:54:22.312410 1594 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 149) took 3129 ms. Trace:
I20250114 02:54:22.312711 1594 rpcz_store.cc:276] 0114 02:54:19.183432 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:19.183843 (+ 411us) service_pool.cc:225] Handling call
0114 02:54:22.309745 (+3125902us) ts_tablet_manager.cc:1145] Deleting tablet db24bf3db61941ba9b5e0662db23e748
0114 02:54:22.309795 (+ 50us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:22.312356 (+ 2561us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3125351}
W20250114 02:54:22.313966 458 catalog_manager.cc:4839] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet db24bf3db61941ba9b5e0662db23e748 because the tablet was not found. No further retry: Not found: Tablet not found: db24bf3db61941ba9b5e0662db23e748
I20250114 02:54:22.323233 1205 tablet_bootstrap.cc:492] T 044832851f804208b0a371eb0857dbf3 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:22.323699 1205 ts_tablet_manager.cc:1397] T 044832851f804208b0a371eb0857dbf3 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.026s user 0.008s sys 0.002s
I20250114 02:54:22.326403 1205 raft_consensus.cc:357] T 044832851f804208b0a371eb0857dbf3 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:22.327220 1205 raft_consensus.cc:383] T 044832851f804208b0a371eb0857dbf3 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:22.327454 1205 raft_consensus.cc:738] T 044832851f804208b0a371eb0857dbf3 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:22.327522 1594 tablet_replica.cc:331] T 7ecf782bb3d54528aa996abca1ea3a15 P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:22.328166 1205 consensus_queue.cc:260] T 044832851f804208b0a371eb0857dbf3 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:22.342139 1702 ts_tablet_manager.cc:1918] T 9812754c610548d1b43084bbc94fc3e4 P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:22.342499 1702 log.cc:1198] T 9812754c610548d1b43084bbc94fc3e4 P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/9812754c610548d1b43084bbc94fc3e4
I20250114 02:54:22.343148 1702 ts_tablet_manager.cc:1939] T 9812754c610548d1b43084bbc94fc3e4 P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:22.346186 1205 ts_tablet_manager.cc:1428] T 044832851f804208b0a371eb0857dbf3 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.022s user 0.006s sys 0.000s
I20250114 02:54:22.346956 1205 tablet_bootstrap.cc:492] T f2b6812e8c1d44529d0284be490df648 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:22.353451 1839 consensus_queue.cc:1035] T 53a8bb487ad64157ad3baecf5d866eca P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:22.355778 1849 tablet.cc:1721] T f464eabfbc0945fa8a5583f00c1dab40 P 525198593c8e47afaee54e74267b87e3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250114 02:54:22.358364 1850 tablet.cc:1721] T f464eabfbc0945fa8a5583f00c1dab40 P 90d6c2c58bb54e0eafb365625ac33871: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250114 02:54:22.366034 1205 tablet_bootstrap.cc:654] T f2b6812e8c1d44529d0284be490df648 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:22.372531 1855 tablet.cc:1721] T f464eabfbc0945fa8a5583f00c1dab40 P 819f9e3412e54d44abb11d8b3ca877c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250114 02:54:22.377190 1784 raft_consensus.cc:513] T 266da813b4514329be0ad97d07b7114f P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:22.380107 813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "266da813b4514329be0ad97d07b7114f" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:22.380882 813 raft_consensus.cc:3054] T 266da813b4514329be0ad97d07b7114f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:22.389923 1786 consensus_queue.cc:1035] T 1381e26725174b9c877075cf5444f566 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250114 02:54:22.391216 751 consensus_peers.cc:487] T 9812754c610548d1b43084bbc94fc3e4 P 819f9e3412e54d44abb11d8b3ca877c2 -> Peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Couldn't send request to peer 525198593c8e47afaee54e74267b87e3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9812754c610548d1b43084bbc94fc3e4. This is attempt 1: this message will repeat every 5th retry.
I20250114 02:54:22.395833 1207 tablet_bootstrap.cc:492] T 71bd00f7d8ac41839f36fd73997631ad P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:22.396422 1207 ts_tablet_manager.cc:1397] T 71bd00f7d8ac41839f36fd73997631ad P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.198s user 0.012s sys 0.000s
I20250114 02:54:22.398990 1207 raft_consensus.cc:357] T 71bd00f7d8ac41839f36fd73997631ad P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:22.399935 1206 ts_tablet_manager.cc:1428] T f2b6812e8c1d44529d0284be490df648 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.121s user 0.005s sys 0.000s
I20250114 02:54:22.400902 1206 tablet_bootstrap.cc:492] T d3fb28717c6f4844af84bb817fe7dfc4 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:22.417280 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet 9812754c610548d1b43084bbc94fc3e4 (table default.Table_2217d3b1dcf4455d9103cad045518573 [id=2199e16ab337450ebc7c664318521edf]) successfully deleted
W20250114 02:54:22.417935 456 catalog_manager.cc:4852] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet 6e858a8449444ef999b9bbef612249d3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e858a8449444ef999b9bbef612249d3 already in progress: creating tablet
I20250114 02:54:22.435283 1594 raft_consensus.cc:2238] T 7ecf782bb3d54528aa996abca1ea3a15 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:22.435842 1594 raft_consensus.cc:2267] T 7ecf782bb3d54528aa996abca1ea3a15 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:22.437887 1594 ts_tablet_manager.cc:1905] T 7ecf782bb3d54528aa996abca1ea3a15 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:22.450249 1205 tablet_bootstrap.cc:492] T f2b6812e8c1d44529d0284be490df648 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:22.450848 1205 ts_tablet_manager.cc:1397] T f2b6812e8c1d44529d0284be490df648 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.104s user 0.015s sys 0.003s
I20250114 02:54:22.453312 1205 raft_consensus.cc:357] T f2b6812e8c1d44529d0284be490df648 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:22.453869 1205 raft_consensus.cc:383] T f2b6812e8c1d44529d0284be490df648 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:22.454082 1205 raft_consensus.cc:738] T f2b6812e8c1d44529d0284be490df648 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:22.454648 1205 consensus_queue.cc:260] T f2b6812e8c1d44529d0284be490df648 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:22.469214 463 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet 26328f8e32bc49ed9b224f0e08b9f949
I20250114 02:54:22.470777 788 tablet_service.cc:1514] Processing DeleteTablet for tablet 26328f8e32bc49ed9b224f0e08b9f949 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:21 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:22.473043 933 tablet_service.cc:1514] Processing DeleteTablet for tablet 26328f8e32bc49ed9b224f0e08b9f949 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:21 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:22.473773 1702 tablet_replica.cc:331] T 26328f8e32bc49ed9b224f0e08b9f949 P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:22.474521 1702 raft_consensus.cc:2238] T 26328f8e32bc49ed9b224f0e08b9f949 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:22.474989 1702 raft_consensus.cc:2267] T 26328f8e32bc49ed9b224f0e08b9f949 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:22.477326 1702 ts_tablet_manager.cc:1905] T 26328f8e32bc49ed9b224f0e08b9f949 P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:22.482647 1205 ts_tablet_manager.cc:1428] T f2b6812e8c1d44529d0284be490df648 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.031s user 0.005s sys 0.000s
I20250114 02:54:22.483426 1205 tablet_bootstrap.cc:492] T d3fb28717c6f4844af84bb817fe7dfc4 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:22.487978 1205 tablet_bootstrap.cc:654] T d3fb28717c6f4844af84bb817fe7dfc4 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:22.494338 813 raft_consensus.cc:2463] T 266da813b4514329be0ad97d07b7114f P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1.
I20250114 02:54:22.510847 469 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet aad96f2507974023aaa988e6950feb7a (table default.Table_9676926a50eb42349a2d609b30566e9a [id=f38063ebf94d4084ab0174428503dc29]). Expected version 2 got 0
I20250114 02:54:22.512060 469 catalog_manager.cc:5526] T 321b9ad6d1ac4d1c80c7a01faf2015af P 525198593c8e47afaee54e74267b87e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 525198593c8e47afaee54e74267b87e3 (127.0.107.66). New cstate: current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:22.517872 469 catalog_manager.cc:5526] T 1381e26725174b9c877075cf5444f566 P 525198593c8e47afaee54e74267b87e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 525198593c8e47afaee54e74267b87e3 (127.0.107.66). New cstate: current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:22.519081 469 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 1381e26725174b9c877075cf5444f566 (table default.Table_b624816662024c55a5f5ef5830b20b7c [id=d2ea84ea59ad4b03ad16bb3052a441f4]). Expected version 2 got 0
I20250114 02:54:22.519765 469 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet f464eabfbc0945fa8a5583f00c1dab40 (table default.Table_4fd891b63924429085d78f78cb177f0e [id=d36c71e7562b4ff1bc8be5cf2955000e]). Expected version 1 got 0
I20250114 02:54:22.471873 1085 tablet_service.cc:1514] Processing DeleteTablet for tablet 26328f8e32bc49ed9b224f0e08b9f949 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:21 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:22.477540 1839 raft_consensus.cc:491] T 6448eea21ab2422c97da2494658ded3f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:22.538300 1207 raft_consensus.cc:383] T 71bd00f7d8ac41839f36fd73997631ad P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:22.538383 933 tablet_service.cc:1467] Processing CreateTablet for tablet a83bae91c29b463aa78412ce2e152ade (DEFAULT_TABLE table=default.Table_579071b88792449ab70a258304f3356c [id=2f767d699397451f8d252f45a07a71d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:22.538154 1839 raft_consensus.cc:513] T 6448eea21ab2422c97da2494658ded3f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:22.538686 1207 raft_consensus.cc:738] T 71bd00f7d8ac41839f36fd73997631ad P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:22.539412 933 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a83bae91c29b463aa78412ce2e152ade. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:22.495640 1042 leader_election.cc:304] T 266da813b4514329be0ad97d07b7114f P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:22.540591 1841 raft_consensus.cc:2798] T 266da813b4514329be0ad97d07b7114f P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:22.540961 1841 raft_consensus.cc:695] T 266da813b4514329be0ad97d07b7114f P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Becoming Leader. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Running, Role: LEADER
I20250114 02:54:22.541707 1841 consensus_queue.cc:237] T 266da813b4514329be0ad97d07b7114f P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:22.542196 1085 tablet_service.cc:1467] Processing CreateTablet for tablet a83bae91c29b463aa78412ce2e152ade (DEFAULT_TABLE table=default.Table_579071b88792449ab70a258304f3356c [id=2f767d699397451f8d252f45a07a71d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:22.543077 1085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a83bae91c29b463aa78412ce2e152ade. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:22.543010 788 tablet_service.cc:1467] Processing CreateTablet for tablet a83bae91c29b463aa78412ce2e152ade (DEFAULT_TABLE table=default.Table_579071b88792449ab70a258304f3356c [id=2f767d699397451f8d252f45a07a71d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:22.543473 960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6448eea21ab2422c97da2494658ded3f" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:22.543962 788 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a83bae91c29b463aa78412ce2e152ade. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:22.544024 960 raft_consensus.cc:2463] T 6448eea21ab2422c97da2494658ded3f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 0.
I20250114 02:54:22.545799 1595 tablet_replica.cc:331] T 26328f8e32bc49ed9b224f0e08b9f949 P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:22.546525 1595 raft_consensus.cc:2238] T 26328f8e32bc49ed9b224f0e08b9f949 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Raft consensus shutting down.
I20250114 02:54:22.547353 1595 raft_consensus.cc:2267] T 26328f8e32bc49ed9b224f0e08b9f949 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250114 02:54:22.540489 1042 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.65:39397, user_credentials={real_user=slave}} blocked reactor thread for 44956.9us
I20250114 02:54:22.550153 1040 leader_election.cc:304] T 6448eea21ab2422c97da2494658ded3f P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:22.551056 1820 raft_consensus.cc:2798] T 6448eea21ab2422c97da2494658ded3f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:22.551460 1820 raft_consensus.cc:491] T 6448eea21ab2422c97da2494658ded3f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:22.551793 1820 raft_consensus.cc:3054] T 6448eea21ab2422c97da2494658ded3f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:22.557382 1783 consensus_queue.cc:1035] T 1381e26725174b9c877075cf5444f566 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:22.558348 1206 tablet_bootstrap.cc:654] T d3fb28717c6f4844af84bb817fe7dfc4 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:22.560624 1595 ts_tablet_manager.cc:1905] T 26328f8e32bc49ed9b224f0e08b9f949 P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:22.562145 1594 ts_tablet_manager.cc:1918] T 7ecf782bb3d54528aa996abca1ea3a15 P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:22.540015 1839 leader_election.cc:290] T 6448eea21ab2422c97da2494658ded3f P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:22.568463 1594 log.cc:1198] T 7ecf782bb3d54528aa996abca1ea3a15 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/7ecf782bb3d54528aa996abca1ea3a15
I20250114 02:54:22.569567 1594 ts_tablet_manager.cc:1939] T 7ecf782bb3d54528aa996abca1ea3a15 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:22.573196 1207 consensus_queue.cc:260] T 71bd00f7d8ac41839f36fd73997631ad P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:22.582485 1594 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 150) took 3396 ms. Trace:
I20250114 02:54:22.582857 1594 rpcz_store.cc:276] 0114 02:54:19.185867 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:19.186001 (+ 134us) service_pool.cc:225] Handling call
0114 02:54:22.327346 (+3141345us) ts_tablet_manager.cc:1145] Deleting tablet 7ecf782bb3d54528aa996abca1ea3a15
0114 02:54:22.327395 (+ 49us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:22.446466 (+119071us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:22.561901 (+115435us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:22.581962 (+ 20061us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":106967,"tablet-delete.queue_time_us":3140706}
I20250114 02:54:22.584152 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet 7ecf782bb3d54528aa996abca1ea3a15 (table default.Table_b0cd93e5a498450e84fb0f1ba4be4505 [id=c70751b59bfc4c6fa60d0e8bb40d5f36]) successfully deleted
I20250114 02:54:22.596282 1820 raft_consensus.cc:513] T 6448eea21ab2422c97da2494658ded3f P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:22.598922 1820 leader_election.cc:290] T 6448eea21ab2422c97da2494658ded3f P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Requested vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:22.600368 1594 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 151) took 3402 ms. Trace:
I20250114 02:54:22.601914 960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6448eea21ab2422c97da2494658ded3f" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
I20250114 02:54:22.602433 960 raft_consensus.cc:3054] T 6448eea21ab2422c97da2494658ded3f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:22.601964 1594 rpcz_store.cc:276] 0114 02:54:19.198056 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:19.198195 (+ 139us) service_pool.cc:225] Handling call
0114 02:54:22.584894 (+3386699us) ts_tablet_manager.cc:1145] Deleting tablet 961b6a25a13a4fa2a732a6383dcd8c84
0114 02:54:22.584940 (+ 46us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:22.600281 (+ 15341us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3386131}
I20250114 02:54:22.600493 817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6448eea21ab2422c97da2494658ded3f" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:22.603194 817 raft_consensus.cc:3054] T 6448eea21ab2422c97da2494658ded3f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
W20250114 02:54:22.603560 458 catalog_manager.cc:4839] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet 961b6a25a13a4fa2a732a6383dcd8c84 because the tablet was not found. No further retry: Not found: Tablet not found: 961b6a25a13a4fa2a732a6383dcd8c84
I20250114 02:54:22.607841 1702 ts_tablet_manager.cc:1918] T 26328f8e32bc49ed9b224f0e08b9f949 P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250114 02:54:22.608253 1702 log.cc:1198] T 26328f8e32bc49ed9b224f0e08b9f949 P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/26328f8e32bc49ed9b224f0e08b9f949
I20250114 02:54:22.608916 1702 ts_tablet_manager.cc:1939] T 26328f8e32bc49ed9b224f0e08b9f949 P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:22.610299 817 raft_consensus.cc:2463] T 6448eea21ab2422c97da2494658ded3f P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1.
I20250114 02:54:22.613260 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet 26328f8e32bc49ed9b224f0e08b9f949 (table default.Table_f851d0fbfd4e4095912dbe3a96fd7142 [id=a01ec721fefc41d6bc4009ca0c327fab]) successfully deleted
I20250114 02:54:22.617084 960 raft_consensus.cc:2463] T 6448eea21ab2422c97da2494658ded3f P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1.
I20250114 02:54:22.620085 1042 leader_election.cc:304] T 6448eea21ab2422c97da2494658ded3f P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:22.620996 1820 raft_consensus.cc:2798] T 6448eea21ab2422c97da2494658ded3f P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:22.622268 1820 raft_consensus.cc:695] T 6448eea21ab2422c97da2494658ded3f P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Becoming Leader. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Running, Role: LEADER
I20250114 02:54:22.623096 1820 consensus_queue.cc:237] T 6448eea21ab2422c97da2494658ded3f P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:22.627149 1741 raft_consensus.cc:491] T 0304df55431e42e1a999ed17dc35e6c0 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:22.629657 815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6448eea21ab2422c97da2494658ded3f" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:22.629916 1594 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 154) took 3364 ms. Trace:
I20250114 02:54:22.630225 815 raft_consensus.cc:2371] T 6448eea21ab2422c97da2494658ded3f P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1. Re-sending same reply.
I20250114 02:54:22.630198 1594 rpcz_store.cc:276] 0114 02:54:19.265233 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:19.265371 (+ 138us) service_pool.cc:225] Handling call
0114 02:54:22.605583 (+3340212us) ts_tablet_manager.cc:1145] Deleting tablet 99b8acb1fe594bf3825c7b9121018db6
0114 02:54:22.629656 (+ 24073us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:22.629867 (+ 211us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3339418}
W20250114 02:54:22.631310 458 catalog_manager.cc:4839] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet 99b8acb1fe594bf3825c7b9121018db6 because the tablet was not found. No further retry: Not found: Tablet not found: 99b8acb1fe594bf3825c7b9121018db6
I20250114 02:54:22.632139 1594 tablet_replica.cc:331] T 266da813b4514329be0ad97d07b7114f P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:22.633191 1594 raft_consensus.cc:2238] T 266da813b4514329be0ad97d07b7114f P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:22.633760 1594 raft_consensus.cc:2267] T 266da813b4514329be0ad97d07b7114f P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:22.627728 1741 raft_consensus.cc:513] T 0304df55431e42e1a999ed17dc35e6c0 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:22.636335 1594 ts_tablet_manager.cc:1905] T 266da813b4514329be0ad97d07b7114f P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250114 02:54:22.641160 1040 leader_election.cc:343] T 266da813b4514329be0ad97d07b7114f P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Not found: Tablet not found: 266da813b4514329be0ad97d07b7114f
I20250114 02:54:22.641613 1784 leader_election.cc:290] T 266da813b4514329be0ad97d07b7114f P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Requested vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:22.646071 1205 tablet_bootstrap.cc:492] T d3fb28717c6f4844af84bb817fe7dfc4 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:22.646513 1205 ts_tablet_manager.cc:1397] T d3fb28717c6f4844af84bb817fe7dfc4 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.163s user 0.005s sys 0.020s
I20250114 02:54:22.648628 960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0304df55431e42e1a999ed17dc35e6c0" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
W20250114 02:54:22.650101 751 leader_election.cc:343] T 0304df55431e42e1a999ed17dc35e6c0 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:22.650781 1783 raft_consensus.cc:491] T 0f1ea1608213492c9a216757896ff2a4 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:22.651240 1783 raft_consensus.cc:513] T 0f1ea1608213492c9a216757896ff2a4 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:22.644008 1741 leader_election.cc:290] T 0304df55431e42e1a999ed17dc35e6c0 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:22.652865 1783 leader_election.cc:290] T 0f1ea1608213492c9a216757896ff2a4 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:22.654373 1099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f1ea1608213492c9a216757896ff2a4" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:22.654711 815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f1ea1608213492c9a216757896ff2a4" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:22.655349 815 raft_consensus.cc:2463] T 0f1ea1608213492c9a216757896ff2a4 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:22.655048 1099 raft_consensus.cc:2463] T 0f1ea1608213492c9a216757896ff2a4 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:22.656168 898 leader_election.cc:304] T 0f1ea1608213492c9a216757896ff2a4 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
I20250114 02:54:22.653687 1102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0304df55431e42e1a999ed17dc35e6c0" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:22.657076 1783 raft_consensus.cc:2798] T 0f1ea1608213492c9a216757896ff2a4 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:22.657377 1783 raft_consensus.cc:491] T 0f1ea1608213492c9a216757896ff2a4 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:22.657649 1783 raft_consensus.cc:3054] T 0f1ea1608213492c9a216757896ff2a4 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
W20250114 02:54:22.660495 753 leader_election.cc:343] T 0304df55431e42e1a999ed17dc35e6c0 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:22.660813 753 leader_election.cc:304] T 0304df55431e42e1a999ed17dc35e6c0 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:22.662034 1741 raft_consensus.cc:2743] T 0304df55431e42e1a999ed17dc35e6c0 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:22.670257 1205 raft_consensus.cc:357] T d3fb28717c6f4844af84bb817fe7dfc4 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:22.671059 1205 raft_consensus.cc:383] T d3fb28717c6f4844af84bb817fe7dfc4 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:22.671356 1205 raft_consensus.cc:738] T d3fb28717c6f4844af84bb817fe7dfc4 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:22.691881 1594 ts_tablet_manager.cc:1918] T 266da813b4514329be0ad97d07b7114f P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:22.693240 933 tablet_service.cc:1467] Processing CreateTablet for tablet 5d545ac27a7f49faace0655713405cea (DEFAULT_TABLE table=default.Table_d101e66d79a64b6f9233687a3cb522a3 [id=491b34b03ce14bd9be16e04915468928]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:22.694252 933 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d545ac27a7f49faace0655713405cea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:22.696818 936 tablet_service.cc:1467] Processing CreateTablet for tablet 0b8361cd538046e0bcb4aac821748f67 (DEFAULT_TABLE table=default.Table_2850a159f211445988d540f3a56ec551 [id=441702f585ce4600be174d46b8607686]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:22.696820 1081 tablet_service.cc:1467] Processing CreateTablet for tablet 5d545ac27a7f49faace0655713405cea (DEFAULT_TABLE table=default.Table_d101e66d79a64b6f9233687a3cb522a3 [id=491b34b03ce14bd9be16e04915468928]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:22.697752 1081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d545ac27a7f49faace0655713405cea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:22.697752 936 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b8361cd538046e0bcb4aac821748f67. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:22.700774 1084 tablet_service.cc:1467] Processing CreateTablet for tablet 0b8361cd538046e0bcb4aac821748f67 (DEFAULT_TABLE table=default.Table_2850a159f211445988d540f3a56ec551 [id=441702f585ce4600be174d46b8607686]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:22.701671 1084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b8361cd538046e0bcb4aac821748f67. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:22.716938 1207 ts_tablet_manager.cc:1428] T 71bd00f7d8ac41839f36fd73997631ad P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.320s user 0.006s sys 0.000s
I20250114 02:54:22.718266 1207 tablet_bootstrap.cc:492] T 0304df55431e42e1a999ed17dc35e6c0 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:22.666932 1783 raft_consensus.cc:513] T 0f1ea1608213492c9a216757896ff2a4 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:22.672044 1205 consensus_queue.cc:260] T d3fb28717c6f4844af84bb817fe7dfc4 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:22.745776 471 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet c86562688d5b4fffb71cbdf7be0afc37
I20250114 02:54:22.747313 471 catalog_manager.cc:6596] ReplaceTablet: tablet c86562688d5b4fffb71cbdf7be0afc37 deleted and replaced by tablet e7ca0bd1dbd542de9ee11e6c72522ae0
I20250114 02:54:22.747931 1880 raft_consensus.cc:491] T 71bd00f7d8ac41839f36fd73997631ad P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:22.748399 1880 raft_consensus.cc:513] T 71bd00f7d8ac41839f36fd73997631ad P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:22.750468 1880 leader_election.cc:290] T 71bd00f7d8ac41839f36fd73997631ad P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:22.750918 941 tablet_service.cc:1514] Processing DeleteTablet for tablet c86562688d5b4fffb71cbdf7be0afc37 with delete_type TABLET_DATA_DELETED (replaced by tablet e7ca0bd1dbd542de9ee11e6c72522ae0) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:22.760219 1849 tablet.cc:1721] T 1381e26725174b9c877075cf5444f566 P 525198593c8e47afaee54e74267b87e3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250114 02:54:22.762714 1850 tablet.cc:1721] T 1381e26725174b9c877075cf5444f566 P 90d6c2c58bb54e0eafb365625ac33871: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250114 02:54:22.695649 788 tablet_service.cc:1467] Processing CreateTablet for tablet 5d545ac27a7f49faace0655713405cea (DEFAULT_TABLE table=default.Table_d101e66d79a64b6f9233687a3cb522a3 [id=491b34b03ce14bd9be16e04915468928]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:22.765551 1594 log.cc:1198] T 266da813b4514329be0ad97d07b7114f P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/266da813b4514329be0ad97d07b7114f
I20250114 02:54:22.766125 788 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d545ac27a7f49faace0655713405cea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:22.766214 1594 ts_tablet_manager.cc:1939] T 266da813b4514329be0ad97d07b7114f P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:22.768179 1594 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 160) took 2994 ms. Trace:
I20250114 02:54:22.768489 1594 rpcz_store.cc:276] 0114 02:54:19.773809 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:19.773926 (+ 117us) service_pool.cc:225] Handling call
0114 02:54:22.631966 (+2858040us) ts_tablet_manager.cc:1145] Deleting tablet 266da813b4514329be0ad97d07b7114f
0114 02:54:22.632011 (+ 45us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:22.668785 (+ 36774us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:22.691689 (+ 22904us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:22.768122 (+ 76433us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2857607}
I20250114 02:54:22.769555 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet 266da813b4514329be0ad97d07b7114f (table default.Table_ae220acb7e0345bdbb80b11398e4d153 [id=bcf9e834733d49a8bc235cd303a75a35]) successfully deleted
I20250114 02:54:22.771449 1099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71bd00f7d8ac41839f36fd73997631ad" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:22.772047 1099 raft_consensus.cc:2463] T 71bd00f7d8ac41839f36fd73997631ad P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:22.772316 790 tablet_service.cc:1467] Processing CreateTablet for tablet 0b8361cd538046e0bcb4aac821748f67 (DEFAULT_TABLE table=default.Table_2850a159f211445988d540f3a56ec551 [id=441702f585ce4600be174d46b8607686]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:22.773373 790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b8361cd538046e0bcb4aac821748f67. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:22.776804 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71bd00f7d8ac41839f36fd73997631ad" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:22.777360 818 raft_consensus.cc:2463] T 71bd00f7d8ac41839f36fd73997631ad P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:22.778127 796 tablet_service.cc:1467] Processing CreateTablet for tablet be7aad41ccdd4ef9bcfe8a985155708f (DEFAULT_TABLE table=default.Table_3e7cad29b0c0447ab9539b45ee482232 [id=f65443bbd53c47e184a6d2e9e99dd1b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:22.779058 796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be7aad41ccdd4ef9bcfe8a985155708f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:22.779147 939 tablet_service.cc:1467] Processing CreateTablet for tablet 612987f3e4ea4a02932ceefa0d740425 (DEFAULT_TABLE table=default.Table_63cdf90ef15a453c9acb19962262eb35 [id=985cc4adbb8a463db0b6a5a5dd49b0ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:22.724858 815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f1ea1608213492c9a216757896ff2a4" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:22.750770 789 tablet_service.cc:1514] Processing DeleteTablet for tablet c86562688d5b4fffb71cbdf7be0afc37 with delete_type TABLET_DATA_DELETED (replaced by tablet e7ca0bd1dbd542de9ee11e6c72522ae0) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:22.780303 941 tablet_service.cc:1467] Processing CreateTablet for tablet be7aad41ccdd4ef9bcfe8a985155708f (DEFAULT_TABLE table=default.Table_3e7cad29b0c0447ab9539b45ee482232 [id=f65443bbd53c47e184a6d2e9e99dd1b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:22.780622 815 raft_consensus.cc:3054] T 0f1ea1608213492c9a216757896ff2a4 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:22.780982 797 tablet_service.cc:1467] Processing CreateTablet for tablet 612987f3e4ea4a02932ceefa0d740425 (DEFAULT_TABLE table=default.Table_63cdf90ef15a453c9acb19962262eb35 [id=985cc4adbb8a463db0b6a5a5dd49b0ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:22.781857 797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 612987f3e4ea4a02932ceefa0d740425. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:22.788514 1084 tablet_service.cc:1514] Processing DeleteTablet for tablet c86562688d5b4fffb71cbdf7be0afc37 with delete_type TABLET_DATA_DELETED (replaced by tablet e7ca0bd1dbd542de9ee11e6c72522ae0) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:22.789611 1081 tablet_service.cc:1467] Processing CreateTablet for tablet be7aad41ccdd4ef9bcfe8a985155708f (DEFAULT_TABLE table=default.Table_3e7cad29b0c0447ab9539b45ee482232 [id=f65443bbd53c47e184a6d2e9e99dd1b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:22.790210 898 leader_election.cc:304] T 71bd00f7d8ac41839f36fd73997631ad P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:22.790835 1786 raft_consensus.cc:2798] T 71bd00f7d8ac41839f36fd73997631ad P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:22.791121 1786 raft_consensus.cc:491] T 71bd00f7d8ac41839f36fd73997631ad P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:22.791442 1786 raft_consensus.cc:3054] T 71bd00f7d8ac41839f36fd73997631ad P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:22.796052 815 raft_consensus.cc:2463] T 0f1ea1608213492c9a216757896ff2a4 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:22.801056 1085 tablet_service.cc:1467] Processing CreateTablet for tablet 612987f3e4ea4a02932ceefa0d740425 (DEFAULT_TABLE table=default.Table_63cdf90ef15a453c9acb19962262eb35 [id=985cc4adbb8a463db0b6a5a5dd49b0ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:22.801829 1820 consensus_queue.cc:1035] T 55b68586ef8b45439a7dd8dcd2efcd79 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:22.806684 1594 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 172) took 2405 ms. Trace:
I20250114 02:54:22.807010 1594 rpcz_store.cc:276] 0114 02:54:20.400907 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:20.401024 (+ 117us) service_pool.cc:225] Handling call
0114 02:54:22.806292 (+2405268us) ts_tablet_manager.cc:1145] Deleting tablet 46360f726fad4534a2daf67581346fc1
0114 02:54:22.806355 (+ 63us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:22.806631 (+ 276us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2404625}
W20250114 02:54:22.808081 458 catalog_manager.cc:4852] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet 46360f726fad4534a2daf67581346fc1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46360f726fad4534a2daf67581346fc1 already in progress: creating tablet
I20250114 02:54:22.811710 1102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f1ea1608213492c9a216757896ff2a4" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:22.812280 1102 raft_consensus.cc:3054] T 0f1ea1608213492c9a216757896ff2a4 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:22.817201 898 leader_election.cc:304] T 0f1ea1608213492c9a216757896ff2a4 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
I20250114 02:54:22.817957 1880 raft_consensus.cc:2798] T 0f1ea1608213492c9a216757896ff2a4 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:22.821957 1102 raft_consensus.cc:2463] T 0f1ea1608213492c9a216757896ff2a4 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:22.823226 1206 tablet_bootstrap.cc:492] T d3fb28717c6f4844af84bb817fe7dfc4 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:22.823740 1206 ts_tablet_manager.cc:1397] T d3fb28717c6f4844af84bb817fe7dfc4 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.423s user 0.006s sys 0.004s
I20250114 02:54:22.827785 1081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be7aad41ccdd4ef9bcfe8a985155708f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:22.833330 1085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 612987f3e4ea4a02932ceefa0d740425. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:22.833292 1880 raft_consensus.cc:695] T 0f1ea1608213492c9a216757896ff2a4 P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Becoming Leader. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Running, Role: LEADER
I20250114 02:54:22.834161 1786 raft_consensus.cc:513] T 71bd00f7d8ac41839f36fd73997631ad P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:22.834105 1880 consensus_queue.cc:237] T 0f1ea1608213492c9a216757896ff2a4 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:22.841815 1883 raft_consensus.cc:491] T 044832851f804208b0a371eb0857dbf3 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:22.842381 1883 raft_consensus.cc:513] T 044832851f804208b0a371eb0857dbf3 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:22.846827 1786 leader_election.cc:290] T 71bd00f7d8ac41839f36fd73997631ad P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Requested vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:22.848143 1099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71bd00f7d8ac41839f36fd73997631ad" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:22.848832 1099 raft_consensus.cc:3054] T 71bd00f7d8ac41839f36fd73997631ad P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:22.851646 1206 raft_consensus.cc:357] T d3fb28717c6f4844af84bb817fe7dfc4 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:22.852538 1206 raft_consensus.cc:383] T d3fb28717c6f4844af84bb817fe7dfc4 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:22.852840 1206 raft_consensus.cc:738] T d3fb28717c6f4844af84bb817fe7dfc4 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:22.854928 463 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet 6448eea21ab2422c97da2494658ded3f (table default.Table_0931c459ae794d2091fcc0f51d031548 [id=e279c31bd63b40b9ade03a3b928bf153]). Expected version 2 got 0
I20250114 02:54:22.858057 1783 leader_election.cc:290] T 0f1ea1608213492c9a216757896ff2a4 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Requested vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:22.859686 788 tablet_service.cc:1467] Processing CreateTablet for tablet a2d68eccaaa64511a9ae36f2b07a7b59 (DEFAULT_TABLE table=default.Table_123095518d164d3db7fe3bd50af9dbd4 [id=4744eda447ca4785a9c065aca94135e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:22.861478 937 tablet_service.cc:1467] Processing CreateTablet for tablet a2d68eccaaa64511a9ae36f2b07a7b59 (DEFAULT_TABLE table=default.Table_123095518d164d3db7fe3bd50af9dbd4 [id=4744eda447ca4785a9c065aca94135e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:22.861739 1207 tablet_bootstrap.cc:654] T 0304df55431e42e1a999ed17dc35e6c0 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:22.865602 463 catalog_manager.cc:5526] T 55b68586ef8b45439a7dd8dcd2efcd79 P 90d6c2c58bb54e0eafb365625ac33871 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67). New cstate: current_term: 1 leader_uuid: "90d6c2c58bb54e0eafb365625ac33871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } }
I20250114 02:54:22.867260 463 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet 55b68586ef8b45439a7dd8dcd2efcd79 (table default.Table_66f2263162f64250a0b49073abe4bd4c [id=c73ede1a5f664d8eb204fb9a012f8f7a]). Expected version 1 got 0
I20250114 02:54:22.870087 1084 tablet_service.cc:1467] Processing CreateTablet for tablet a2d68eccaaa64511a9ae36f2b07a7b59 (DEFAULT_TABLE table=default.Table_123095518d164d3db7fe3bd50af9dbd4 [id=4744eda447ca4785a9c065aca94135e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:22.871189 1084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2d68eccaaa64511a9ae36f2b07a7b59. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:22.871045 789 tablet_service.cc:1467] Processing CreateTablet for tablet 670dc7ff15d644e18c296eeb2a6b99e1 (DEFAULT_TABLE table=default.Table_fd43493446ca49a48e8a970bb92ea239 [id=6dbd2db73b5443a2af5d3b5056ab0169]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:22.878952 939 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 612987f3e4ea4a02932ceefa0d740425. 1 dirs total, 0 dirs full, 0 dirs failed
W20250114 02:54:22.879599 456 catalog_manager.cc:4852] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet c86562688d5b4fffb71cbdf7be0afc37 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c86562688d5b4fffb71cbdf7be0afc37 already in progress: creating tablet
I20250114 02:54:22.879840 937 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2d68eccaaa64511a9ae36f2b07a7b59. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:22.875954 1206 consensus_queue.cc:260] T d3fb28717c6f4844af84bb817fe7dfc4 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:22.884948 1883 leader_election.cc:290] T 044832851f804208b0a371eb0857dbf3 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:22.885325 934 tablet_service.cc:1467] Processing CreateTablet for tablet 670dc7ff15d644e18c296eeb2a6b99e1 (DEFAULT_TABLE table=default.Table_fd43493446ca49a48e8a970bb92ea239 [id=6dbd2db73b5443a2af5d3b5056ab0169]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:22.885592 941 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be7aad41ccdd4ef9bcfe8a985155708f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:22.891716 1083 tablet_service.cc:1467] Processing CreateTablet for tablet 670dc7ff15d644e18c296eeb2a6b99e1 (DEFAULT_TABLE table=default.Table_fd43493446ca49a48e8a970bb92ea239 [id=6dbd2db73b5443a2af5d3b5056ab0169]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:22.892693 1083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 670dc7ff15d644e18c296eeb2a6b99e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:22.896544 934 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 670dc7ff15d644e18c296eeb2a6b99e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:22.898976 1820 consensus_queue.cc:1035] T 55b68586ef8b45439a7dd8dcd2efcd79 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:54:22.908962 933 tablet_service.cc:1467] Processing CreateTablet for tablet 68981913851243a7a1875fc99efb3c49 (DEFAULT_TABLE table=default.Table_e2ae3006a8ca4082aee1af910f435789 [id=683a61c5af09453e8027e9cb5b5b80a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:22.919745 463 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet 1381e26725174b9c877075cf5444f566 (table default.Table_b624816662024c55a5f5ef5830b20b7c [id=d2ea84ea59ad4b03ad16bb3052a441f4]). Expected version 2 got 0
I20250114 02:54:22.917809 1207 tablet_bootstrap.cc:492] T 0304df55431e42e1a999ed17dc35e6c0 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:22.920897 463 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet f464eabfbc0945fa8a5583f00c1dab40 (table default.Table_4fd891b63924429085d78f78cb177f0e [id=d36c71e7562b4ff1bc8be5cf2955000e]). Expected version 1 got 0
I20250114 02:54:22.921764 1207 ts_tablet_manager.cc:1397] T 0304df55431e42e1a999ed17dc35e6c0 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.204s user 0.012s sys 0.000s
I20250114 02:54:22.926005 933 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68981913851243a7a1875fc99efb3c49. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:22.925007 793 tablet_service.cc:1467] Processing CreateTablet for tablet 68981913851243a7a1875fc99efb3c49 (DEFAULT_TABLE table=default.Table_e2ae3006a8ca4082aee1af910f435789 [id=683a61c5af09453e8027e9cb5b5b80a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:22.894608 1206 ts_tablet_manager.cc:1428] T d3fb28717c6f4844af84bb817fe7dfc4 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.063s user 0.004s sys 0.002s
I20250114 02:54:22.930518 1101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "044832851f804208b0a371eb0857dbf3" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:22.934290 942 tablet_service.cc:1467] Processing CreateTablet for tablet d56255ca806d4e2cac09fbb5e1b1efbd (DEFAULT_TABLE table=default.Table_cec05793ad284a18a7da3557696f5bcb [id=55c4d259157546c58ce29dd45f979ab2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:22.939240 942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d56255ca806d4e2cac09fbb5e1b1efbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:22.939993 1080 tablet_service.cc:1467] Processing CreateTablet for tablet d56255ca806d4e2cac09fbb5e1b1efbd (DEFAULT_TABLE table=default.Table_cec05793ad284a18a7da3557696f5bcb [id=55c4d259157546c58ce29dd45f979ab2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:22.941443 1080 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d56255ca806d4e2cac09fbb5e1b1efbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:22.950008 816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "044832851f804208b0a371eb0857dbf3" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:22.950703 816 raft_consensus.cc:2463] T 044832851f804208b0a371eb0857dbf3 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:22.952252 1877 raft_consensus.cc:491] T f2b6812e8c1d44529d0284be490df648 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:22.952749 1877 raft_consensus.cc:513] T f2b6812e8c1d44529d0284be490df648 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:22.954730 1877 leader_election.cc:290] T f2b6812e8c1d44529d0284be490df648 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
W20250114 02:54:22.956805 458 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet a52d6749ff834b62a8e20a2f45935240 (table default.Table_1c9b7896941e4289b1c12e20c210c2c1 [id=b3ee15b9f10f4d489edbde3af9034d25], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet a52d6749ff834b62a8e20a2f45935240 consensus state has no leader
I20250114 02:54:22.959486 1101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2b6812e8c1d44529d0284be490df648" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
W20250114 02:54:22.961251 753 leader_election.cc:343] T f2b6812e8c1d44529d0284be490df648 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:22.961915 814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71bd00f7d8ac41839f36fd73997631ad" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:22.962499 814 raft_consensus.cc:3054] T 71bd00f7d8ac41839f36fd73997631ad P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:22.968499 1888 tablet.cc:1721] T 1381e26725174b9c877075cf5444f566 P 819f9e3412e54d44abb11d8b3ca877c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250114 02:54:22.972416 814 raft_consensus.cc:2463] T 71bd00f7d8ac41839f36fd73997631ad P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:22.975378 1594 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 173) took 2433 ms. Trace:
I20250114 02:54:22.975685 1594 rpcz_store.cc:276] 0114 02:54:20.542255 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:20.596296 (+ 54041us) service_pool.cc:225] Handling call
0114 02:54:22.826275 (+2229979us) ts_tablet_manager.cc:1145] Deleting tablet ca986f81ed6848a89588366f3f400eec
0114 02:54:22.975006 (+148731us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:22.975325 (+ 319us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2229450}
W20250114 02:54:22.977038 1042 consensus_peers.cc:487] T bc46815b4803418c95ea9c039a4fae88 P 90d6c2c58bb54e0eafb365625ac33871 -> Peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Couldn't send request to peer 819f9e3412e54d44abb11d8b3ca877c2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bc46815b4803418c95ea9c039a4fae88. This is attempt 1: this message will repeat every 5th retry.
W20250114 02:54:22.977955 458 catalog_manager.cc:4852] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet ca986f81ed6848a89588366f3f400eec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca986f81ed6848a89588366f3f400eec already in progress: creating tablet
I20250114 02:54:22.984818 789 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 670dc7ff15d644e18c296eeb2a6b99e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:22.985460 793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68981913851243a7a1875fc99efb3c49. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:22.985566 1084 tablet_service.cc:1467] Processing CreateTablet for tablet 68981913851243a7a1875fc99efb3c49 (DEFAULT_TABLE table=default.Table_e2ae3006a8ca4082aee1af910f435789 [id=683a61c5af09453e8027e9cb5b5b80a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:22.986523 1084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68981913851243a7a1875fc99efb3c49. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:22.989136 1207 raft_consensus.cc:357] T 0304df55431e42e1a999ed17dc35e6c0 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:22.992929 1207 raft_consensus.cc:383] T 0304df55431e42e1a999ed17dc35e6c0 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:22.993232 959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2b6812e8c1d44529d0284be490df648" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:22.993404 898 leader_election.cc:304] T 044832851f804208b0a371eb0857dbf3 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
I20250114 02:54:22.993813 959 raft_consensus.cc:2463] T f2b6812e8c1d44529d0284be490df648 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 0.
I20250114 02:54:22.994029 788 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2d68eccaaa64511a9ae36f2b07a7b59. 1 dirs total, 0 dirs full, 0 dirs failed
W20250114 02:54:23.004547 455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.107.124:39105: connect: Connection refused (error 111) [suppressed 6 similar messages]
I20250114 02:54:23.008725 1205 ts_tablet_manager.cc:1428] T d3fb28717c6f4844af84bb817fe7dfc4 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.362s user 0.006s sys 0.001s
I20250114 02:54:23.009667 1205 tablet_bootstrap.cc:492] T ca986f81ed6848a89588366f3f400eec P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:23.020426 1207 raft_consensus.cc:738] T 0304df55431e42e1a999ed17dc35e6c0 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:23.021973 789 tablet_service.cc:1467] Processing CreateTablet for tablet d56255ca806d4e2cac09fbb5e1b1efbd (DEFAULT_TABLE table=default.Table_cec05793ad284a18a7da3557696f5bcb [id=55c4d259157546c58ce29dd45f979ab2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:23.021569 1207 consensus_queue.cc:260] T 0304df55431e42e1a999ed17dc35e6c0 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:23.013229 1099 raft_consensus.cc:2463] T 71bd00f7d8ac41839f36fd73997631ad P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:23.030261 470 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet 97f4e061d3b94ee9982410b7ce135453
I20250114 02:54:23.031967 470 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:33540 (request call id 221) took 1286 ms. Trace:
I20250114 02:54:23.033967 1207 ts_tablet_manager.cc:1428] T 0304df55431e42e1a999ed17dc35e6c0 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.111s user 0.005s sys 0.000s
I20250114 02:54:23.033995 1086 tablet_service.cc:1514] Processing DeleteTablet for tablet 97f4e061d3b94ee9982410b7ce135453 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:21 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:23.036572 1595 ts_tablet_manager.cc:1918] T 26328f8e32bc49ed9b224f0e08b9f949 P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250114 02:54:23.037103 1595 log.cc:1198] T 26328f8e32bc49ed9b224f0e08b9f949 P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/26328f8e32bc49ed9b224f0e08b9f949
I20250114 02:54:23.037144 790 tablet_service.cc:1514] Processing DeleteTablet for tablet 97f4e061d3b94ee9982410b7ce135453 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:21 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:23.038084 1595 ts_tablet_manager.cc:1939] T 26328f8e32bc49ed9b224f0e08b9f949 P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
I20250114 02:54:23.034004 470 rpcz_store.cc:276] 0114 02:54:21.745080 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:21.880118 (+135038us) service_pool.cc:225] Handling call
0114 02:54:21.881617 (+ 1499us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0114 02:54:21.882270 (+ 653us) catalog_manager.cc:2753] Modifying in-memory table state
0114 02:54:21.882419 (+ 149us) catalog_manager.cc:2762] Locking tablets
0114 02:54:22.527235 (+644816us) catalog_manager.cc:2776] Removing table and tablets from system table
0114 02:54:22.645830 (+118595us) catalog_manager.cc:2794] Removing table from by-name map
0114 02:54:22.646094 (+ 264us) catalog_manager.cc:2810] Committing in-memory state
0114 02:54:23.030143 (+384049us) catalog_manager.cc:2814] Aborting table tasks
0114 02:54:23.031894 (+ 1751us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:22.579566 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:22.580013 (+ 447us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:22.580044 (+ 31us) write_op.cc:435] Acquired schema lock
0114 02:54:22.580063 (+ 19us) tablet.cc:659] Decoding operations
0114 02:54:22.580452 (+ 389us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:22.580499 (+ 47us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:22.580531 (+ 32us) tablet.cc:682] Acquiring locks for 2 operations
0114 02:54:22.580853 (+ 322us) tablet.cc:698] Row locks acquired
0114 02:54:22.580872 (+ 19us) write_op.cc:260] PREPARE: finished
0114 02:54:22.581000 (+ 128us) write_op.cc:270] Start()
0114 02:54:22.581097 (+ 97us) write_op.cc:276] Timestamp: P: 1736823262580921 usec, L: 0
0114 02:54:22.581117 (+ 20us) op_driver.cc:348] REPLICATION: starting
0114 02:54:22.581678 (+ 561us) log.cc:844] Serialized 728 byte log entry
0114 02:54:22.637068 (+ 55390us) op_driver.cc:464] REPLICATION: finished
0114 02:54:22.639196 (+ 2128us) write_op.cc:301] APPLY: starting
0114 02:54:22.639261 (+ 65us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:22.639385 (+ 124us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:22.639491 (+ 106us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:22.639750 (+ 259us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:22.639906 (+ 156us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0114 02:54:22.639959 (+ 53us) write_op.cc:312] APPLY: finished
0114 02:54:22.640133 (+ 174us) log.cc:844] Serialized 29 byte log entry
0114 02:54:22.645305 (+ 5172us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:22.645401 (+ 96us) write_op.cc:454] Released schema lock
0114 02:54:22.645445 (+ 44us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1984,"apply.run_cpu_time_us":1468,"apply.run_wall_time_us":6451,"mutex_wait_us":49813,"num_ops":2,"prepare.queue_time_us":50208,"prepare.run_cpu_time_us":2482,"prepare.run_wall_time_us":40911,"raft.queue_time_us":1239,"raft.run_cpu_time_us":10,"raft.run_wall_time_us":10,"replication_time_us":55717,"thread_start_us":72751,"threads_started":1}]]}
I20250114 02:54:23.040962 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet 26328f8e32bc49ed9b224f0e08b9f949 (table default.Table_f851d0fbfd4e4095912dbe3a96fd7142 [id=a01ec721fefc41d6bc4009ca0c327fab]) successfully deleted
I20250114 02:54:23.045315 1207 tablet_bootstrap.cc:492] T ca986f81ed6848a89588366f3f400eec P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
W20250114 02:54:23.045778 458 catalog_manager.cc:4852] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet c86562688d5b4fffb71cbdf7be0afc37 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c86562688d5b4fffb71cbdf7be0afc37 already in progress: creating tablet
I20250114 02:54:23.060540 1594 tablet_replica.cc:331] T a456420f78714cf895528e19dfe0ee8d P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:23.061044 789 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d56255ca806d4e2cac09fbb5e1b1efbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:23.061316 1594 raft_consensus.cc:2238] T a456420f78714cf895528e19dfe0ee8d P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 LEADER]: Raft consensus shutting down.
I20250114 02:54:23.061650 1206 tablet_bootstrap.cc:492] T ca986f81ed6848a89588366f3f400eec P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:23.062196 1594 raft_consensus.cc:2267] T a456420f78714cf895528e19dfe0ee8d P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:23.064129 1594 ts_tablet_manager.cc:1905] T a456420f78714cf895528e19dfe0ee8d P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:23.073735 933 tablet_service.cc:1514] Processing DeleteTablet for tablet 97f4e061d3b94ee9982410b7ce135453 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:21 UTC) from {username='slave'} at 127.0.0.1:55068
W20250114 02:54:23.075522 456 catalog_manager.cc:4852] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet 97f4e061d3b94ee9982410b7ce135453 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 97f4e061d3b94ee9982410b7ce135453 already in progress: creating tablet
I20250114 02:54:23.080238 1207 tablet_bootstrap.cc:654] T ca986f81ed6848a89588366f3f400eec P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:23.090917 1784 consensus_queue.cc:1035] T bc46815b4803418c95ea9c039a4fae88 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:23.097288 1880 raft_consensus.cc:2798] T 044832851f804208b0a371eb0857dbf3 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:23.097663 1880 raft_consensus.cc:491] T 044832851f804208b0a371eb0857dbf3 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:23.097971 1880 raft_consensus.cc:3054] T 044832851f804208b0a371eb0857dbf3 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:23.101293 471 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 55b68586ef8b45439a7dd8dcd2efcd79 (table default.Table_66f2263162f64250a0b49073abe4bd4c [id=c73ede1a5f664d8eb204fb9a012f8f7a]). Expected version 1 got 0
I20250114 02:54:23.102624 471 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 1381e26725174b9c877075cf5444f566 (table default.Table_b624816662024c55a5f5ef5830b20b7c [id=d2ea84ea59ad4b03ad16bb3052a441f4]). Expected version 2 got 0
I20250114 02:54:23.103544 1594 ts_tablet_manager.cc:1918] T a456420f78714cf895528e19dfe0ee8d P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:23.103873 1594 log.cc:1198] T a456420f78714cf895528e19dfe0ee8d P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/a456420f78714cf895528e19dfe0ee8d
I20250114 02:54:23.104583 1594 ts_tablet_manager.cc:1939] T a456420f78714cf895528e19dfe0ee8d P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:23.104315 471 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 0304df55431e42e1a999ed17dc35e6c0 (table default.Table_4c3843233dda407395ab9652d5870bc6 [id=0d173fdb502f482fb86b6d3e5fdc66f1]). Expected version 1 got 0
I20250114 02:54:23.105090 471 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 6448eea21ab2422c97da2494658ded3f (table default.Table_0931c459ae794d2091fcc0f51d031548 [id=e279c31bd63b40b9ade03a3b928bf153]). Expected version 2 got 0
I20250114 02:54:23.105602 471 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 044832851f804208b0a371eb0857dbf3 (table default.Table_0518d8bed7304c86aff32e05e6f07c0d [id=515a2fe3f1e84e82bccc02ef5cecef1b]). Expected version 1 got 0
I20250114 02:54:23.109609 467 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: default.Table_3139140b7eec4849ae715c295fbeaa6b [id=8536eb7f6bf542daa88a8db0e4873456] alter complete (version 2)
I20250114 02:54:23.109998 1207 tablet_bootstrap.cc:492] T ca986f81ed6848a89588366f3f400eec P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:23.110384 1207 ts_tablet_manager.cc:1397] T ca986f81ed6848a89588366f3f400eec P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.065s user 0.006s sys 0.003s
I20250114 02:54:23.109969 1880 raft_consensus.cc:513] T 044832851f804208b0a371eb0857dbf3 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:23.111670 1880 leader_election.cc:290] T 044832851f804208b0a371eb0857dbf3 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Requested vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
W20250114 02:54:23.112851 898 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.65:39397, user_credentials={real_user=slave}} blocked reactor thread for 119525us
I20250114 02:54:23.113760 898 leader_election.cc:304] T 71bd00f7d8ac41839f36fd73997631ad P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
I20250114 02:54:23.115933 1207 raft_consensus.cc:357] T ca986f81ed6848a89588366f3f400eec P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:23.116712 1207 raft_consensus.cc:383] T ca986f81ed6848a89588366f3f400eec P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:23.116940 1207 raft_consensus.cc:738] T ca986f81ed6848a89588366f3f400eec P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:23.121240 1786 raft_consensus.cc:2798] T 71bd00f7d8ac41839f36fd73997631ad P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:23.121642 1594 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 174) took 2085 ms. Trace:
I20250114 02:54:23.121655 1786 raft_consensus.cc:695] T 71bd00f7d8ac41839f36fd73997631ad P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Becoming Leader. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Running, Role: LEADER
I20250114 02:54:23.121964 1594 rpcz_store.cc:276] 0114 02:54:21.036608 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:21.036785 (+ 177us) service_pool.cc:225] Handling call
0114 02:54:22.978021 (+1941236us) ts_tablet_manager.cc:1145] Deleting tablet a456420f78714cf895528e19dfe0ee8d
0114 02:54:23.060405 (+ 82384us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:23.073197 (+ 12792us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:23.103332 (+ 30135us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:23.121574 (+ 18242us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1940488}
I20250114 02:54:23.122442 1786 consensus_queue.cc:237] T 71bd00f7d8ac41839f36fd73997631ad P 525198593c8e47afaee54e74267b87e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:23.123196 1207 consensus_queue.cc:260] T ca986f81ed6848a89588366f3f400eec P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:23.127005 1207 ts_tablet_manager.cc:1428] T ca986f81ed6848a89588366f3f400eec P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
W20250114 02:54:23.127830 898 leader_election.cc:343] T 044832851f804208b0a371eb0857dbf3 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:23.131951 751 leader_election.cc:304] T f2b6812e8c1d44529d0284be490df648 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters: 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:23.132912 1877 raft_consensus.cc:2798] T f2b6812e8c1d44529d0284be490df648 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:23.133200 1877 raft_consensus.cc:491] T f2b6812e8c1d44529d0284be490df648 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:23.133493 1877 raft_consensus.cc:3054] T f2b6812e8c1d44529d0284be490df648 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:23.136736 1594 tablet_replica.cc:331] T 9812754c610548d1b43084bbc94fc3e4 P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:23.140655 1877 raft_consensus.cc:513] T f2b6812e8c1d44529d0284be490df648 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:23.141886 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet a456420f78714cf895528e19dfe0ee8d (table default.Table_fd43493446ca49a48e8a970bb92ea239 [id=6dbd2db73b5443a2af5d3b5056ab0169]) successfully deleted
I20250114 02:54:23.144552 1102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2b6812e8c1d44529d0284be490df648" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
W20250114 02:54:23.145777 753 leader_election.cc:343] T f2b6812e8c1d44529d0284be490df648 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:23.147251 1877 leader_election.cc:290] T f2b6812e8c1d44529d0284be490df648 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Requested vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:23.149410 1207 tablet_bootstrap.cc:492] T d3fb28717c6f4844af84bb817fe7dfc4 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:23.153201 939 tablet_service.cc:1514] Processing DeleteTablet for tablet 290ad07ab8be4791a39194909e000843 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:13 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:23.154441 1594 raft_consensus.cc:2238] T 9812754c610548d1b43084bbc94fc3e4 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 LEADER]: Raft consensus shutting down.
W20250114 02:54:23.154536 456 catalog_manager.cc:4839] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet 290ad07ab8be4791a39194909e000843 because the tablet was not found. No further retry: Not found: Tablet not found: 290ad07ab8be4791a39194909e000843
I20250114 02:54:23.155309 1594 raft_consensus.cc:2267] T 9812754c610548d1b43084bbc94fc3e4 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:23.156167 939 tablet_service.cc:1514] Processing DeleteTablet for tablet 99b8acb1fe594bf3825c7b9121018db6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:16 UTC) from {username='slave'} at 127.0.0.1:55068
W20250114 02:54:23.157490 456 catalog_manager.cc:4839] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet 99b8acb1fe594bf3825c7b9121018db6 because the tablet was not found. No further retry: Not found: Tablet not found: 99b8acb1fe594bf3825c7b9121018db6
W20250114 02:54:23.159884 467 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.66:50663 (request call id 43) took 3880 ms (client timeout 1000 ms). Trace:
I20250114 02:54:23.165232 817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "044832851f804208b0a371eb0857dbf3" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:23.165897 817 raft_consensus.cc:3054] T 044832851f804208b0a371eb0857dbf3 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:23.170006 1594 ts_tablet_manager.cc:1905] T 9812754c610548d1b43084bbc94fc3e4 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:23.175099 936 tablet_service.cc:1514] Processing DeleteTablet for tablet 9812754c610548d1b43084bbc94fc3e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:19 UTC) from {username='slave'} at 127.0.0.1:55068
W20250114 02:54:23.177085 456 catalog_manager.cc:4839] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet 9812754c610548d1b43084bbc94fc3e4 because the tablet was not found. No further retry: Not found: Tablet not found: 9812754c610548d1b43084bbc94fc3e4
I20250114 02:54:23.178385 1207 tablet_bootstrap.cc:654] T d3fb28717c6f4844af84bb817fe7dfc4 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:23.182281 939 tablet_service.cc:1514] Processing DeleteTablet for tablet ba215eb8422d4f879dd0bf5996543962 with delete_type TABLET_DATA_DELETED (replaced by tablet f40c66314342453b898da3ac0e7e86bf) from {username='slave'} at 127.0.0.1:55068
W20250114 02:54:23.183895 458 catalog_manager.cc:4852] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 97f4e061d3b94ee9982410b7ce135453 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 97f4e061d3b94ee9982410b7ce135453 already in progress: creating tablet
I20250114 02:54:23.189268 1102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "044832851f804208b0a371eb0857dbf3" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
W20250114 02:54:23.194334 456 catalog_manager.cc:4839] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet ba215eb8422d4f879dd0bf5996543962 because the tablet was not found. No further retry: Not found: Tablet not found: ba215eb8422d4f879dd0bf5996543962
I20250114 02:54:23.201359 817 raft_consensus.cc:2463] T 044832851f804208b0a371eb0857dbf3 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:23.203853 1205 tablet_bootstrap.cc:654] T ca986f81ed6848a89588366f3f400eec P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:23.203909 1594 ts_tablet_manager.cc:1918] T 9812754c610548d1b43084bbc94fc3e4 P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:23.204646 960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2b6812e8c1d44529d0284be490df648" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
I20250114 02:54:23.205210 960 raft_consensus.cc:3054] T f2b6812e8c1d44529d0284be490df648 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:23.206741 1207 tablet_bootstrap.cc:492] T d3fb28717c6f4844af84bb817fe7dfc4 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:23.207142 1207 ts_tablet_manager.cc:1397] T d3fb28717c6f4844af84bb817fe7dfc4 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.058s user 0.011s sys 0.000s
I20250114 02:54:23.209729 1207 raft_consensus.cc:357] T d3fb28717c6f4844af84bb817fe7dfc4 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:23.210407 1207 raft_consensus.cc:383] T d3fb28717c6f4844af84bb817fe7dfc4 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:23.210655 1207 raft_consensus.cc:738] T d3fb28717c6f4844af84bb817fe7dfc4 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:23.211236 1207 consensus_queue.cc:260] T d3fb28717c6f4844af84bb817fe7dfc4 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:23.211680 960 raft_consensus.cc:2463] T f2b6812e8c1d44529d0284be490df648 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 1.
I20250114 02:54:23.224277 898 leader_election.cc:304] T 044832851f804208b0a371eb0857dbf3 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
I20250114 02:54:23.225217 1880 raft_consensus.cc:2798] T 044832851f804208b0a371eb0857dbf3 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:23.224274 468 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: default.Table_ec0eefb64458440ebd57a05a7ad0bb9a [id=b6874c22fe0d4151a7c8ea061f685938] alter complete (version 1)
I20250114 02:54:23.226361 1877 raft_consensus.cc:491] T d3fb28717c6f4844af84bb817fe7dfc4 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:23.225646 1880 raft_consensus.cc:695] T 044832851f804208b0a371eb0857dbf3 P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Becoming Leader. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Running, Role: LEADER
I20250114 02:54:23.226787 1877 raft_consensus.cc:513] T d3fb28717c6f4844af84bb817fe7dfc4 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:23.231328 1877 leader_election.cc:290] T d3fb28717c6f4844af84bb817fe7dfc4 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:23.232105 1102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3fb28717c6f4844af84bb817fe7dfc4" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:23.232792 1102 raft_consensus.cc:2463] T d3fb28717c6f4844af84bb817fe7dfc4 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 0.
I20250114 02:54:23.235148 753 leader_election.cc:304] T d3fb28717c6f4844af84bb817fe7dfc4 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:23.236002 1877 raft_consensus.cc:2798] T d3fb28717c6f4844af84bb817fe7dfc4 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:23.236354 1877 raft_consensus.cc:491] T d3fb28717c6f4844af84bb817fe7dfc4 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:23.236672 1877 raft_consensus.cc:3054] T d3fb28717c6f4844af84bb817fe7dfc4 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:23.227632 1880 consensus_queue.cc:237] T 044832851f804208b0a371eb0857dbf3 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:23.242447 1594 log.cc:1198] T 9812754c610548d1b43084bbc94fc3e4 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/9812754c610548d1b43084bbc94fc3e4
I20250114 02:54:23.243219 1594 ts_tablet_manager.cc:1939] T 9812754c610548d1b43084bbc94fc3e4 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:23.245549 1207 ts_tablet_manager.cc:1428] T d3fb28717c6f4844af84bb817fe7dfc4 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.038s user 0.006s sys 0.000s
I20250114 02:54:23.250233 1084 tablet_service.cc:1514] Processing DeleteTablet for tablet 290ad07ab8be4791a39194909e000843 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:13 UTC) from {username='slave'} at 127.0.0.1:53876
W20250114 02:54:23.254170 462 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.65:49805 (request call id 46) took 2915 ms (client timeout 1000 ms). Trace:
I20250114 02:54:23.255048 1877 raft_consensus.cc:513] T d3fb28717c6f4844af84bb817fe7dfc4 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:23.257370 788 tablet_service.cc:1514] Processing DeleteTablet for tablet 66bfa92db9874750b3c340e31a06c412 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:16 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:23.258026 789 tablet_service.cc:1514] Processing DeleteTablet for tablet ba215eb8422d4f879dd0bf5996543962 with delete_type TABLET_DATA_DELETED (replaced by tablet f40c66314342453b898da3ac0e7e86bf) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:23.263615 1081 tablet_service.cc:1514] Processing DeleteTablet for tablet 00080bea6c8e46e584ccba6dcb0e221b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:17 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:23.268111 793 tablet_service.cc:1514] Processing DeleteTablet for tablet 266da813b4514329be0ad97d07b7114f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:19 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:23.276582 1594 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 182) took 1555 ms. Trace:
I20250114 02:54:23.276914 1594 rpcz_store.cc:276] 0114 02:54:21.720871 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:21.721147 (+ 276us) service_pool.cc:225] Handling call
0114 02:54:23.136575 (+1415428us) ts_tablet_manager.cc:1145] Deleting tablet 9812754c610548d1b43084bbc94fc3e4
0114 02:54:23.136628 (+ 53us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:23.183399 (+ 46771us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:23.203696 (+ 20297us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:23.276528 (+ 72832us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1414366}
I20250114 02:54:23.278015 1206 tablet_bootstrap.cc:654] T ca986f81ed6848a89588366f3f400eec P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:23.278474 1207 tablet_bootstrap.cc:492] T f2b6812e8c1d44529d0284be490df648 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:23.280283 1084 tablet_service.cc:1514] Processing DeleteTablet for tablet 9812754c610548d1b43084bbc94fc3e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:19 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:23.281152 1084 tablet_service.cc:1514] Processing DeleteTablet for tablet ba215eb8422d4f879dd0bf5996543962 with delete_type TABLET_DATA_DELETED (replaced by tablet f40c66314342453b898da3ac0e7e86bf) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:23.283864 1081 tablet_service.cc:1514] Processing DeleteTablet for tablet ec3285525faf482d8739f1bedd653503 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:20 UTC) from {username='slave'} at 127.0.0.1:53876
W20250114 02:54:23.289341 462 rpcz_store.cc:269] 0114 02:54:20.339130 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:20.543438 (+204308us) service_pool.cc:225] Handling call
0114 02:54:23.254069 (+2710631us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":45327,"reported_tablets":14}
I20250114 02:54:23.291344 462 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_77652aa0d7a94f1f98afb881a924eb24" } new_table_name: "default.Table_8f8a18d7868c44bbb4d66a0692b08186" modify_external_catalogs: true
W20250114 02:54:23.260480 465 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.67:34825 (request call id 42) took 3023 ms (client timeout 1000 ms). Trace:
I20250114 02:54:23.294366 1594 tablet_replica.cc:331] T ec3285525faf482d8739f1bedd653503 P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
W20250114 02:54:23.294523 465 rpcz_store.cc:269] 0114 02:54:20.237428 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:20.384279 (+146851us) service_pool.cc:225] Handling call
0114 02:54:20.561681 (+177402us) spinlock_profiling.cc:243] Waited 150 ms on lock 0x7b54000316d4. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616cad50 00007f646174df2a 00007f64633f2dec 00007f64634eadee 00007f6461248fc4 00007f6461248f52 00007f6461248e81 00007f6461248dfc 00007f64612480b2 00007f645f512dfc 00007f645f512236
0114 02:54:23.260394 (+2698713us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16148,"reported_tablets":20,"spinlock_wait_cycles":345627904}
W20250114 02:54:23.296562 898 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.65:39397, user_credentials={real_user=slave}} blocked reactor thread for 71536.4us
I20250114 02:54:23.295101 1594 raft_consensus.cc:2238] T ec3285525faf482d8739f1bedd653503 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:23.297982 1594 raft_consensus.cc:2267] T ec3285525faf482d8739f1bedd653503 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:23.297504 465 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_73ec0be3adc440d788c369d80d7bbefa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:23.267088 466 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.67:34825 (request call id 43) took 2015 ms (client timeout 1000 ms). Trace:
I20250114 02:54:23.304183 1102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3fb28717c6f4844af84bb817fe7dfc4" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:23.304736 1102 raft_consensus.cc:3054] T d3fb28717c6f4844af84bb817fe7dfc4 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
W20250114 02:54:23.304370 466 rpcz_store.cc:269] 0114 02:54:21.251714 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:21.294292 (+ 42578us) service_pool.cc:225] Handling call
0114 02:54:21.304792 (+ 10500us) spinlock_profiling.cc:243] Waited 6.06 ms on lock 0x7b54000316d4. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616cad50 00007f646174df2a 00007f64633f2dec 00007f64634eadee 00007f6461248fc4 00007f6461248f52 00007f6461248e81 00007f6461248dfc 00007f64612480b2 00007f645f512dfc 00007f645f512236
0114 02:54:21.588879 (+284087us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:21.588915 (+ 36us) write_op.cc:454] Released schema lock
0114 02:54:23.267000 (+1678085us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:21.571007 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:21.571358 (+ 351us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:21.571379 (+ 21us) write_op.cc:435] Acquired schema lock
0114 02:54:21.571392 (+ 13us) tablet.cc:659] Decoding operations
0114 02:54:21.571632 (+ 240us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:21.571668 (+ 36us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:21.571689 (+ 21us) tablet.cc:682] Acquiring locks for 1 operations
0114 02:54:21.571822 (+ 133us) tablet.cc:698] Row locks acquired
0114 02:54:21.571835 (+ 13us) write_op.cc:260] PREPARE: finished
0114 02:54:21.571933 (+ 98us) write_op.cc:270] Start()
0114 02:54:21.572007 (+ 74us) write_op.cc:276] Timestamp: P: 1736823261571863 usec, L: 0
0114 02:54:21.572021 (+ 14us) op_driver.cc:348] REPLICATION: starting
0114 02:54:21.572476 (+ 455us) log.cc:844] Serialized 530 byte log entry
0114 02:54:21.579874 (+ 7398us) op_driver.cc:464] REPLICATION: finished
0114 02:54:21.580270 (+ 396us) write_op.cc:301] APPLY: starting
0114 02:54:21.580333 (+ 63us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:21.580426 (+ 93us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:21.580504 (+ 78us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:21.580660 (+ 156us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:21.580771 (+ 111us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:21.580809 (+ 38us) write_op.cc:312] APPLY: finished
0114 02:54:21.580994 (+ 185us) log.cc:844] Serialized 23 byte log entry
0114 02:54:21.581118 (+ 124us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:21.581172 (+ 54us) write_op.cc:454] Released schema lock
0114 02:54:21.581199 (+ 27us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":22,"spinlock_wait_cycles":13938432,"child_traces":[["op",{"apply.queue_time_us":247,"apply.run_cpu_time_us":1094,"apply.run_wall_time_us":1094,"num_ops":1,"prepare.queue_time_us":3260,"prepare.run_cpu_time_us":1738,"prepare.run_wall_time_us":16854,"raft.queue_time_us":75,"raft.run_cpu_time_us":582,"raft.run_wall_time_us":583,"replication_time_us":7769}]]}
I20250114 02:54:23.312067 1102 raft_consensus.cc:2463] T d3fb28717c6f4844af84bb817fe7dfc4 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 1.
I20250114 02:54:23.313031 753 leader_election.cc:304] T d3fb28717c6f4844af84bb817fe7dfc4 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:23.313753 1741 raft_consensus.cc:2798] T d3fb28717c6f4844af84bb817fe7dfc4 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:23.319219 1877 leader_election.cc:290] T d3fb28717c6f4844af84bb817fe7dfc4 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Requested vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:23.320636 1786 raft_consensus.cc:491] T d3fb28717c6f4844af84bb817fe7dfc4 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:23.321034 1786 raft_consensus.cc:513] T d3fb28717c6f4844af84bb817fe7dfc4 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:23.323240 1207 tablet_bootstrap.cc:654] T f2b6812e8c1d44529d0284be490df648 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:23.323297 1786 leader_election.cc:290] T d3fb28717c6f4844af84bb817fe7dfc4 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:23.329205 1594 ts_tablet_manager.cc:1905] T ec3285525faf482d8739f1bedd653503 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:23.330952 936 tablet_service.cc:1514] Processing DeleteTablet for tablet 53a8bb487ad64157ad3baecf5d866eca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:17 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:23.331820 933 tablet_service.cc:1514] Processing DeleteTablet for tablet bc46815b4803418c95ea9c039a4fae88 with delete_type TABLET_DATA_DELETED (replaced by tablet 97f4e061d3b94ee9982410b7ce135453) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:23.331862 1702 tablet_replica.cc:331] T 53a8bb487ad64157ad3baecf5d866eca P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:23.331815 941 tablet_service.cc:1514] Processing DeleteTablet for tablet 266da813b4514329be0ad97d07b7114f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:19 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:23.333032 934 tablet_service.cc:1514] Processing DeleteTablet for tablet 9812754c610548d1b43084bbc94fc3e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:19 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:23.333294 942 tablet_service.cc:1514] Processing DeleteTablet for tablet 66bfa92db9874750b3c340e31a06c412 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:16 UTC) from {username='slave'} at 127.0.0.1:55068
W20250114 02:54:23.335184 898 leader_election.cc:343] T 044832851f804208b0a371eb0857dbf3 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:23.336119 1085 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ecf782bb3d54528aa996abca1ea3a15 with delete_type TABLET_DATA_DELETED (replaced by tablet 66bfa92db9874750b3c340e31a06c412) from {username='slave'} at 127.0.0.1:53876
W20250114 02:54:23.262351 458 catalog_manager.cc:4839] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 290ad07ab8be4791a39194909e000843 because the tablet was not found. No further retry: Not found: Tablet not found: 290ad07ab8be4791a39194909e000843
I20250114 02:54:23.349053 1741 raft_consensus.cc:695] T d3fb28717c6f4844af84bb817fe7dfc4 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 LEADER]: Becoming Leader. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Running, Role: LEADER
W20250114 02:54:23.349576 458 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.107.67:45749, user_credentials={real_user=slave}} blocked reactor thread for 87240.4us
I20250114 02:54:23.350060 1741 consensus_queue.cc:237] T d3fb28717c6f4844af84bb817fe7dfc4 P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:23.352176 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet 9812754c610548d1b43084bbc94fc3e4 (table default.Table_2217d3b1dcf4455d9103cad045518573 [id=2199e16ab337450ebc7c664318521edf]) successfully deleted
W20250114 02:54:23.353215 458 catalog_manager.cc:4839] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 00080bea6c8e46e584ccba6dcb0e221b because the tablet was not found. No further retry: Not found: Tablet not found: 00080bea6c8e46e584ccba6dcb0e221b
I20250114 02:54:23.353307 751 leader_election.cc:304] T f2b6812e8c1d44529d0284be490df648 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters: 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:23.354230 1899 raft_consensus.cc:2798] T f2b6812e8c1d44529d0284be490df648 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:23.354638 1899 raft_consensus.cc:695] T f2b6812e8c1d44529d0284be490df648 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 LEADER]: Becoming Leader. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Running, Role: LEADER
I20250114 02:54:23.355662 1081 tablet_service.cc:1514] Processing DeleteTablet for tablet 53a8bb487ad64157ad3baecf5d866eca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:17 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:23.355756 1102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3fb28717c6f4844af84bb817fe7dfc4" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:23.355932 817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3fb28717c6f4844af84bb817fe7dfc4" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:23.355326 1899 consensus_queue.cc:237] T f2b6812e8c1d44529d0284be490df648 P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:23.357460 960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3fb28717c6f4844af84bb817fe7dfc4" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
W20250114 02:54:23.354211 458 catalog_manager.cc:4839] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 9812754c610548d1b43084bbc94fc3e4 because the tablet was not found. No further retry: Not found: Tablet not found: 9812754c610548d1b43084bbc94fc3e4
W20250114 02:54:23.358913 458 catalog_manager.cc:4839] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet ba215eb8422d4f879dd0bf5996543962 because the tablet was not found. No further retry: Not found: Tablet not found: ba215eb8422d4f879dd0bf5996543962
W20250114 02:54:23.359598 458 catalog_manager.cc:4839] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet ec3285525faf482d8739f1bedd653503 because the tablet was not found. No further retry: Not found: Tablet not found: ec3285525faf482d8739f1bedd653503
I20250114 02:54:23.358120 960 raft_consensus.cc:2463] T d3fb28717c6f4844af84bb817fe7dfc4 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 0.
W20250114 02:54:23.360275 458 catalog_manager.cc:4839] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 7ecf782bb3d54528aa996abca1ea3a15 because the tablet was not found. No further retry: Not found: Tablet not found: 7ecf782bb3d54528aa996abca1ea3a15
W20250114 02:54:23.160349 467 rpcz_store.cc:269] 0114 02:54:19.279078 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:19.279386 (+ 308us) service_pool.cc:225] Handling call
0114 02:54:23.159801 (+3880415us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:21.047641 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:21.047980 (+ 339us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:21.048002 (+ 22us) write_op.cc:435] Acquired schema lock
0114 02:54:21.048015 (+ 13us) tablet.cc:659] Decoding operations
0114 02:54:21.056994 (+ 8979us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:21.057035 (+ 41us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:21.057059 (+ 24us) tablet.cc:682] Acquiring locks for 1 operations
0114 02:54:21.057210 (+ 151us) tablet.cc:698] Row locks acquired
0114 02:54:21.057222 (+ 12us) write_op.cc:260] PREPARE: finished
0114 02:54:21.057341 (+ 119us) write_op.cc:270] Start()
0114 02:54:21.057494 (+ 153us) write_op.cc:276] Timestamp: P: 1736823261057255 usec, L: 0
0114 02:54:21.057512 (+ 18us) op_driver.cc:348] REPLICATION: starting
0114 02:54:21.057978 (+ 466us) log.cc:844] Serialized 530 byte log entry
0114 02:54:21.081081 (+ 23103us) op_driver.cc:464] REPLICATION: finished
0114 02:54:21.081445 (+ 364us) write_op.cc:301] APPLY: starting
0114 02:54:21.081497 (+ 52us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:21.081584 (+ 87us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:21.081669 (+ 85us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:21.081819 (+ 150us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:21.081935 (+ 116us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:21.081970 (+ 35us) write_op.cc:312] APPLY: finished
0114 02:54:21.082143 (+ 173us) log.cc:844] Serialized 23 byte log entry
0114 02:54:21.082275 (+ 132us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:21.082341 (+ 66us) write_op.cc:454] Released schema lock
0114 02:54:21.082367 (+ 26us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:21.124801 (+ 42434us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:21.124836 (+ 35us) write_op.cc:454] Released schema lock
Related trace 'op':
0114 02:54:21.278737 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:21.279071 (+ 334us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:21.279092 (+ 21us) write_op.cc:435] Acquired schema lock
0114 02:54:21.279105 (+ 13us) tablet.cc:659] Decoding operations
0114 02:54:21.279360 (+ 255us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:21.279392 (+ 32us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:21.279413 (+ 21us) tablet.cc:682] Acquiring locks for 1 operations
0114 02:54:21.279535 (+ 122us) tablet.cc:698] Row locks acquired
0114 02:54:21.279548 (+ 13us) write_op.cc:260] PREPARE: finished
0114 02:54:21.279643 (+ 95us) write_op.cc:270] Start()
0114 02:54:21.280056 (+ 413us) write_op.cc:276] Timestamp: P: 1736823261279602 usec, L: 0
0114 02:54:21.280071 (+ 15us) op_driver.cc:348] REPLICATION: starting
0114 02:54:21.281374 (+ 1303us) log.cc:844] Serialized 356 byte log entry
0114 02:54:21.301601 (+ 20227us) op_driver.cc:464] REPLICATION: finished
0114 02:54:21.302184 (+ 583us) write_op.cc:301] APPLY: starting
0114 02:54:21.302234 (+ 50us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:21.302331 (+ 97us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:21.302413 (+ 82us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:21.302549 (+ 136us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:21.302673 (+ 124us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:21.302710 (+ 37us) write_op.cc:312] APPLY: finished
0114 02:54:21.302913 (+ 203us) log.cc:844] Serialized 23 byte log entry
0114 02:54:21.303043 (+ 130us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:21.303122 (+ 79us) write_op.cc:454] Released schema lock
0114 02:54:21.303151 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:21.309068 (+ 5917us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:21.309094 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":152266,"reported_tablets":9,"child_traces":[["op",{"apply.queue_time_us":221,"apply.run_cpu_time_us":1092,"apply.run_wall_time_us":1092,"num_ops":1,"prepare.queue_time_us":171054,"prepare.run_cpu_time_us":1984,"prepare.run_wall_time_us":77129,"raft.queue_time_us":146,"raft.run_cpu_time_us":712,"raft.run_wall_time_us":711,"replication_time_us":23480}],["op",{"apply.queue_time_us":457,"apply.run_cpu_time_us":1134,"apply.run_wall_time_us":1134,"num_ops":1,"prepare.queue_time_us":193368,"prepare.run_cpu_time_us":1817,"prepare.run_wall_time_us":30313,"raft.queue_time_us":67,"raft.run_cpu_time_us":495,"raft.run_wall_time_us":495,"replication_time_us":21447}]]}
I20250114 02:54:23.358145 1102 raft_consensus.cc:2388] T d3fb28717c6f4844af84bb817fe7dfc4 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 525198593c8e47afaee54e74267b87e3 in current term 1: Already voted for candidate 819f9e3412e54d44abb11d8b3ca877c2 in this term.
W20250114 02:54:23.274291 464 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 6448eea21ab2422c97da2494658ded3f (table default.Table_0931c459ae794d2091fcc0f51d031548 [id=e279c31bd63b40b9ade03a3b928bf153], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet 6448eea21ab2422c97da2494658ded3f consensus state has no leader
I20250114 02:54:23.357121 1595 tablet_replica.cc:331] T 53a8bb487ad64157ad3baecf5d866eca P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
W20250114 02:54:23.365365 464 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.65:49805 (request call id 47) took 1976 ms (client timeout 1000 ms). Trace:
I20250114 02:54:23.365634 1595 raft_consensus.cc:2238] T 53a8bb487ad64157ad3baecf5d866eca P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Raft consensus shutting down.
W20250114 02:54:23.365779 464 rpcz_store.cc:269] 0114 02:54:21.388640 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:21.455760 (+ 67120us) service_pool.cc:225] Handling call
0114 02:54:21.509654 (+ 53894us) spinlock_profiling.cc:243] Waited 50.7 ms on lock 0x7b54000316d4. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616cad50 00007f646174df2a 00007f64633f2dec 00007f64634eadee 00007f6461248fc4 00007f6461248f52 00007f6461248e81 00007f6461248dfc 00007f64612480b2 00007f645f512dfc 00007f645f512236
0114 02:54:23.365284 (+1855630us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15926,"reported_tablets":15,"spinlock_wait_cycles":116630912}
I20250114 02:54:23.366504 1595 raft_consensus.cc:2267] T 53a8bb487ad64157ad3baecf5d866eca P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250114 02:54:23.326994 469 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.66:50663 (request call id 45) took 1753 ms (client timeout 1000 ms). Trace:
I20250114 02:54:23.368855 1595 ts_tablet_manager.cc:1905] T 53a8bb487ad64157ad3baecf5d866eca P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250114 02:54:23.368606 469 rpcz_store.cc:269] 0114 02:54:21.573510 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:21.623799 (+ 50289us) service_pool.cc:225] Handling call
0114 02:54:22.684472 (+1060673us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:22.684510 (+ 38us) write_op.cc:454] Released schema lock
0114 02:54:23.326889 (+642379us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:22.575440 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:22.575910 (+ 470us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:22.575944 (+ 34us) write_op.cc:435] Acquired schema lock
0114 02:54:22.575964 (+ 20us) tablet.cc:659] Decoding operations
0114 02:54:22.576387 (+ 423us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:22.576441 (+ 54us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:22.576475 (+ 34us) tablet.cc:682] Acquiring locks for 2 operations
0114 02:54:22.576817 (+ 342us) tablet.cc:698] Row locks acquired
0114 02:54:22.576835 (+ 18us) write_op.cc:260] PREPARE: finished
0114 02:54:22.576977 (+ 142us) write_op.cc:270] Start()
0114 02:54:22.577082 (+ 105us) write_op.cc:276] Timestamp: P: 1736823262576896 usec, L: 0
0114 02:54:22.577103 (+ 21us) op_driver.cc:348] REPLICATION: starting
0114 02:54:22.577721 (+ 618us) log.cc:844] Serialized 907 byte log entry
0114 02:54:22.636625 (+ 58904us) op_driver.cc:464] REPLICATION: finished
0114 02:54:22.637590 (+ 965us) write_op.cc:301] APPLY: starting
0114 02:54:22.637657 (+ 67us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:22.637794 (+ 137us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:22.637914 (+ 120us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:22.638180 (+ 266us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:22.638342 (+ 162us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0114 02:54:22.638405 (+ 63us) write_op.cc:312] APPLY: finished
0114 02:54:22.638609 (+ 204us) log.cc:844] Serialized 29 byte log entry
0114 02:54:22.638764 (+ 155us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:22.638843 (+ 79us) write_op.cc:454] Released schema lock
0114 02:54:22.638882 (+ 39us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":102321,"reported_tablets":16,"child_traces":[["op",{"apply.queue_time_us":775,"apply.run_cpu_time_us":1480,"apply.run_wall_time_us":1480,"num_ops":2,"prepare.queue_time_us":53022,"prepare.run_cpu_time_us":2621,"prepare.run_wall_time_us":4008,"raft.queue_time_us":9318,"raft.run_cpu_time_us":1004,"raft.run_wall_time_us":1004,"replication_time_us":59400}]]}
I20250114 02:54:23.376441 1085 tablet_service.cc:1514] Processing DeleteTablet for tablet 66bfa92db9874750b3c340e31a06c412 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:16 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:23.381330 1702 raft_consensus.cc:2238] T 53a8bb487ad64157ad3baecf5d866eca P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:23.381800 1702 raft_consensus.cc:2267] T 53a8bb487ad64157ad3baecf5d866eca P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250114 02:54:23.384337 1040 consensus_peers.cc:487] T 266da813b4514329be0ad97d07b7114f P 90d6c2c58bb54e0eafb365625ac33871 -> Peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Couldn't send request to peer 525198593c8e47afaee54e74267b87e3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 266da813b4514329be0ad97d07b7114f. This is attempt 1: this message will repeat every 5th retry.
I20250114 02:54:23.385356 1702 ts_tablet_manager.cc:1905] T 53a8bb487ad64157ad3baecf5d866eca P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250114 02:54:23.299784 465 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_73ec0be3adc440d788c369d80d7bbefa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:23.379634 466 master_service.cc:946] ReplaceTablet: received request to replace tablet e15e6e52d7294aeeb4463b3a76022ec3 from {username='slave'} at 127.0.0.1:33540
I20250114 02:54:23.391638 1207 tablet_bootstrap.cc:492] T f2b6812e8c1d44529d0284be490df648 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:23.392207 1207 ts_tablet_manager.cc:1397] T f2b6812e8c1d44529d0284be490df648 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.114s user 0.006s sys 0.006s
I20250114 02:54:23.400913 1207 raft_consensus.cc:357] T f2b6812e8c1d44529d0284be490df648 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:23.406270 1207 raft_consensus.cc:383] T f2b6812e8c1d44529d0284be490df648 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:23.408411 1207 raft_consensus.cc:738] T f2b6812e8c1d44529d0284be490df648 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:23.413328 1207 consensus_queue.cc:260] T f2b6812e8c1d44529d0284be490df648 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:23.427874 960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3fb28717c6f4844af84bb817fe7dfc4" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
I20250114 02:54:23.428571 960 raft_consensus.cc:3054] T d3fb28717c6f4844af84bb817fe7dfc4 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:23.435217 960 raft_consensus.cc:2463] T d3fb28717c6f4844af84bb817fe7dfc4 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 1.
I20250114 02:54:23.451882 1207 ts_tablet_manager.cc:1428] T f2b6812e8c1d44529d0284be490df648 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.059s user 0.007s sys 0.000s
I20250114 02:54:23.454634 1207 tablet_bootstrap.cc:492] T 044832851f804208b0a371eb0857dbf3 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:23.465065 1206 tablet_bootstrap.cc:492] T ca986f81ed6848a89588366f3f400eec P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:23.465569 1206 ts_tablet_manager.cc:1397] T ca986f81ed6848a89588366f3f400eec P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.404s user 0.005s sys 0.005s
I20250114 02:54:23.468621 464 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_651d4cc08fe74c4aa0bbcf876d11b8e8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:23.470587 464 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_651d4cc08fe74c4aa0bbcf876d11b8e8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:23.474929 469 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_ebe5f78fc2dc4adeacda49c55c5b2147"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:23.477094 469 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ebe5f78fc2dc4adeacda49c55c5b2147 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:23.460918 1205 tablet_bootstrap.cc:492] T ca986f81ed6848a89588366f3f400eec P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:23.478027 1205 ts_tablet_manager.cc:1397] T ca986f81ed6848a89588366f3f400eec P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.469s user 0.008s sys 0.004s
I20250114 02:54:23.480298 1205 raft_consensus.cc:357] T ca986f81ed6848a89588366f3f400eec P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:23.480993 1205 raft_consensus.cc:383] T ca986f81ed6848a89588366f3f400eec P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:23.481299 1205 raft_consensus.cc:738] T ca986f81ed6848a89588366f3f400eec P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:23.481648 1702 ts_tablet_manager.cc:1918] T 53a8bb487ad64157ad3baecf5d866eca P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:23.481998 1702 log.cc:1198] T 53a8bb487ad64157ad3baecf5d866eca P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/53a8bb487ad64157ad3baecf5d866eca
I20250114 02:54:23.481927 1205 consensus_queue.cc:260] T ca986f81ed6848a89588366f3f400eec P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:23.484367 1205 ts_tablet_manager.cc:1428] T ca986f81ed6848a89588366f3f400eec P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250114 02:54:23.466792 898 leader_election.cc:304] T d3fb28717c6f4844af84bb817fe7dfc4 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 525198593c8e47afaee54e74267b87e3; no voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:23.486106 1786 raft_consensus.cc:2743] T d3fb28717c6f4844af84bb817fe7dfc4 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:23.487848 1594 ts_tablet_manager.cc:1918] T ec3285525faf482d8739f1bedd653503 P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:23.492887 1206 raft_consensus.cc:357] T ca986f81ed6848a89588366f3f400eec P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:23.493814 1206 raft_consensus.cc:383] T ca986f81ed6848a89588366f3f400eec P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:23.494107 1206 raft_consensus.cc:738] T ca986f81ed6848a89588366f3f400eec P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:23.494824 1206 consensus_queue.cc:260] T ca986f81ed6848a89588366f3f400eec P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:23.497959 1205 tablet_bootstrap.cc:492] T 0304df55431e42e1a999ed17dc35e6c0 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:23.502730 1594 log.cc:1198] T ec3285525faf482d8739f1bedd653503 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/ec3285525faf482d8739f1bedd653503
I20250114 02:54:23.503742 1594 ts_tablet_manager.cc:1939] T ec3285525faf482d8739f1bedd653503 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:23.505751 1594 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 181) took 1785 ms. Trace:
I20250114 02:54:23.515453 1595 ts_tablet_manager.cc:1918] T 53a8bb487ad64157ad3baecf5d866eca P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:23.530864 1594 rpcz_store.cc:276] 0114 02:54:21.720571 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:21.722456 (+ 1885us) service_pool.cc:225] Handling call
0114 02:54:23.294222 (+1571766us) ts_tablet_manager.cc:1145] Deleting tablet ec3285525faf482d8739f1bedd653503
0114 02:54:23.294264 (+ 42us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:23.389500 (+ 95236us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:23.487643 (+ 98143us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:23.505612 (+ 17969us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1571033}
W20250114 02:54:23.528874 1042 consensus_peers.cc:487] T 266da813b4514329be0ad97d07b7114f P 90d6c2c58bb54e0eafb365625ac33871 -> Peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Couldn't send request to peer 819f9e3412e54d44abb11d8b3ca877c2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 266da813b4514329be0ad97d07b7114f. This is attempt 1: this message will repeat every 5th retry.
I20250114 02:54:23.531848 1207 tablet_bootstrap.cc:654] T 044832851f804208b0a371eb0857dbf3 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:23.536741 1595 log.cc:1198] T 53a8bb487ad64157ad3baecf5d866eca P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/53a8bb487ad64157ad3baecf5d866eca
I20250114 02:54:23.538856 1595 ts_tablet_manager.cc:1939] T 53a8bb487ad64157ad3baecf5d866eca P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
I20250114 02:54:23.544555 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet ec3285525faf482d8739f1bedd653503 (table default.Table_4598efdd6ff348d081ed8c3e7dc3130f [id=fdd50384f4894e20b21f109c6a33864c]) successfully deleted
I20250114 02:54:23.545349 1594 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 184) took 1679 ms. Trace:
I20250114 02:54:23.549347 1594 rpcz_store.cc:276] 0114 02:54:21.865593 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:21.869115 (+ 3522us) service_pool.cc:225] Handling call
0114 02:54:23.544699 (+1675584us) ts_tablet_manager.cc:1145] Deleting tablet 6e858a8449444ef999b9bbef612249d3
0114 02:54:23.544741 (+ 42us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:23.545293 (+ 552us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1673656}
I20250114 02:54:23.530565 1206 ts_tablet_manager.cc:1428] T ca986f81ed6848a89588366f3f400eec P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.065s user 0.005s sys 0.000s
W20250114 02:54:23.550549 458 catalog_manager.cc:4852] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet 6e858a8449444ef999b9bbef612249d3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e858a8449444ef999b9bbef612249d3 already in progress: creating tablet
I20250114 02:54:23.482750 1702 ts_tablet_manager.cc:1939] T 53a8bb487ad64157ad3baecf5d866eca P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:23.553596 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet 53a8bb487ad64157ad3baecf5d866eca (table default.Table_5d3b333f1d99463b8ad3b4318c8f29df [id=0f1c410e9c5c40ab8b8fe4e4bf68e8ea]) successfully deleted
I20250114 02:54:23.554154 1205 tablet_bootstrap.cc:654] T 0304df55431e42e1a999ed17dc35e6c0 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:23.563333 1595 tablet_replica.cc:331] T 66bfa92db9874750b3c340e31a06c412 P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:23.566270 1595 raft_consensus.cc:2238] T 66bfa92db9874750b3c340e31a06c412 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Raft consensus shutting down.
I20250114 02:54:23.562785 470 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 55b68586ef8b45439a7dd8dcd2efcd79 (table default.Table_66f2263162f64250a0b49073abe4bd4c [id=c73ede1a5f664d8eb204fb9a012f8f7a]). Expected version 1 got 0
I20250114 02:54:23.567246 1595 raft_consensus.cc:2267] T 66bfa92db9874750b3c340e31a06c412 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:23.568488 1205 tablet_bootstrap.cc:492] T 0304df55431e42e1a999ed17dc35e6c0 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:23.568882 1205 ts_tablet_manager.cc:1397] T 0304df55431e42e1a999ed17dc35e6c0 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.071s user 0.005s sys 0.004s
I20250114 02:54:23.573963 1205 raft_consensus.cc:357] T 0304df55431e42e1a999ed17dc35e6c0 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:23.571213 1207 tablet_bootstrap.cc:492] T 044832851f804208b0a371eb0857dbf3 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:23.574646 1205 raft_consensus.cc:383] T 0304df55431e42e1a999ed17dc35e6c0 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:23.576367 1207 ts_tablet_manager.cc:1397] T 044832851f804208b0a371eb0857dbf3 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.124s user 0.010s sys 0.000s
I20250114 02:54:23.576617 1205 raft_consensus.cc:738] T 0304df55431e42e1a999ed17dc35e6c0 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:23.577220 1205 consensus_queue.cc:260] T 0304df55431e42e1a999ed17dc35e6c0 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:23.579427 1205 ts_tablet_manager.cc:1428] T 0304df55431e42e1a999ed17dc35e6c0 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20250114 02:54:23.580204 1205 tablet_bootstrap.cc:492] T 46360f726fad4534a2daf67581346fc1 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:23.581588 1784 consensus_queue.cc:1035] T 6448eea21ab2422c97da2494658ded3f P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:23.582751 1702 tablet_replica.cc:331] T bc46815b4803418c95ea9c039a4fae88 P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:23.583271 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet 53a8bb487ad64157ad3baecf5d866eca (table default.Table_5d3b333f1d99463b8ad3b4318c8f29df [id=0f1c410e9c5c40ab8b8fe4e4bf68e8ea]) successfully deleted
I20250114 02:54:23.583849 1702 raft_consensus.cc:2238] T bc46815b4803418c95ea9c039a4fae88 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:23.586251 1702 raft_consensus.cc:2267] T bc46815b4803418c95ea9c039a4fae88 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:23.590380 1205 tablet_bootstrap.cc:654] T 46360f726fad4534a2daf67581346fc1 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:23.590852 1702 ts_tablet_manager.cc:1905] T bc46815b4803418c95ea9c039a4fae88 P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:23.593828 1594 tablet_replica.cc:331] T 26328f8e32bc49ed9b224f0e08b9f949 P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:23.594861 1594 raft_consensus.cc:2238] T 26328f8e32bc49ed9b224f0e08b9f949 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:23.595599 1594 raft_consensus.cc:2267] T 26328f8e32bc49ed9b224f0e08b9f949 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:23.599279 1206 tablet_bootstrap.cc:492] T 46360f726fad4534a2daf67581346fc1 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:23.599754 1820 consensus_queue.cc:1035] T 6448eea21ab2422c97da2494658ded3f P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:23.603307 1205 tablet_bootstrap.cc:492] T 46360f726fad4534a2daf67581346fc1 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:23.603708 1205 ts_tablet_manager.cc:1397] T 46360f726fad4534a2daf67581346fc1 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.003s
I20250114 02:54:23.605964 1206 tablet_bootstrap.cc:654] T 46360f726fad4534a2daf67581346fc1 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:23.605895 1205 raft_consensus.cc:357] T 46360f726fad4534a2daf67581346fc1 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:23.606547 1205 raft_consensus.cc:383] T 46360f726fad4534a2daf67581346fc1 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:23.606806 1205 raft_consensus.cc:738] T 46360f726fad4534a2daf67581346fc1 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:23.613085 1205 consensus_queue.cc:260] T 46360f726fad4534a2daf67581346fc1 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:23.615973 1205 ts_tablet_manager.cc:1428] T 46360f726fad4534a2daf67581346fc1 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.012s user 0.005s sys 0.001s
I20250114 02:54:23.618911 1205 tablet_bootstrap.cc:492] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:23.623687 1595 ts_tablet_manager.cc:1905] T 66bfa92db9874750b3c340e31a06c412 P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:23.627462 1205 tablet_bootstrap.cc:654] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:23.627424 1594 ts_tablet_manager.cc:1905] T 26328f8e32bc49ed9b224f0e08b9f949 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:23.630316 1207 raft_consensus.cc:357] T 044832851f804208b0a371eb0857dbf3 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:23.630973 1207 raft_consensus.cc:383] T 044832851f804208b0a371eb0857dbf3 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:23.631229 1207 raft_consensus.cc:738] T 044832851f804208b0a371eb0857dbf3 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:23.649935 1207 consensus_queue.cc:260] T 044832851f804208b0a371eb0857dbf3 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:23.665378 1207 ts_tablet_manager.cc:1428] T 044832851f804208b0a371eb0857dbf3 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.088s user 0.005s sys 0.000s
I20250114 02:54:23.666241 1207 tablet_bootstrap.cc:492] T 46360f726fad4534a2daf67581346fc1 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:23.670223 470 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 1381e26725174b9c877075cf5444f566 (table default.Table_b624816662024c55a5f5ef5830b20b7c [id=d2ea84ea59ad4b03ad16bb3052a441f4]). Expected version 2 got 0
I20250114 02:54:23.671635 470 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 6448eea21ab2422c97da2494658ded3f (table default.Table_0931c459ae794d2091fcc0f51d031548 [id=e279c31bd63b40b9ade03a3b928bf153]). Expected version 2 got 0
I20250114 02:54:23.700099 1784 raft_consensus.cc:491] T 0304df55431e42e1a999ed17dc35e6c0 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:23.700677 1784 raft_consensus.cc:513] T 0304df55431e42e1a999ed17dc35e6c0 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:23.702153 1207 tablet_bootstrap.cc:654] T 46360f726fad4534a2daf67581346fc1 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:23.702983 1784 leader_election.cc:290] T 0304df55431e42e1a999ed17dc35e6c0 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:23.705052 814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0304df55431e42e1a999ed17dc35e6c0" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:23.705948 814 raft_consensus.cc:2463] T 0304df55431e42e1a999ed17dc35e6c0 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 0.
I20250114 02:54:23.706979 1042 leader_election.cc:304] T 0304df55431e42e1a999ed17dc35e6c0 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:23.707631 1784 raft_consensus.cc:2798] T 0304df55431e42e1a999ed17dc35e6c0 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:23.707940 1784 raft_consensus.cc:491] T 0304df55431e42e1a999ed17dc35e6c0 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:23.708300 1784 raft_consensus.cc:3054] T 0304df55431e42e1a999ed17dc35e6c0 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:23.708146 960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0304df55431e42e1a999ed17dc35e6c0" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:23.708755 960 raft_consensus.cc:2463] T 0304df55431e42e1a999ed17dc35e6c0 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 0.
I20250114 02:54:23.708724 469 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33540 (request call id 231) took 1432 ms. Trace:
I20250114 02:54:23.705477 1206 tablet_bootstrap.cc:492] T 46360f726fad4534a2daf67581346fc1 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:23.710659 1206 ts_tablet_manager.cc:1397] T 46360f726fad4534a2daf67581346fc1 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.112s user 0.007s sys 0.003s
W20250114 02:54:23.712811 462 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 719188f639724851a5fc2b37cb60fdbb consensus state has no leader
I20250114 02:54:23.713374 462 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:33540 (request call id 227) took 1677 ms. Trace:
I20250114 02:54:23.719197 466 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet e15e6e52d7294aeeb4463b3a76022ec3
I20250114 02:54:23.722229 793 tablet_service.cc:1514] Processing DeleteTablet for tablet e15e6e52d7294aeeb4463b3a76022ec3 with delete_type TABLET_DATA_DELETED (replaced by tablet 6d741bfd00784d2d830be3ebe50dc7c4) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:23.722436 939 tablet_service.cc:1514] Processing DeleteTablet for tablet e15e6e52d7294aeeb4463b3a76022ec3 with delete_type TABLET_DATA_DELETED (replaced by tablet 6d741bfd00784d2d830be3ebe50dc7c4) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:23.724013 1884 consensus_queue.cc:1035] T 0f1ea1608213492c9a216757896ff2a4 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:54:23.726007 465 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33540 (request call id 228) took 1575 ms. Trace:
I20250114 02:54:23.709111 469 rpcz_store.cc:276] 0114 02:54:22.276195 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:23.386469 (+1110274us) service_pool.cc:225] Handling call
0114 02:54:23.474723 (+ 88254us) spinlock_profiling.cc:243] Waited 87 ms on lock 0x7b54000316d4. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616cad50 00007f646174df2a 00007f64633f2dec 00007f64634ec306 00007f6461250044 00007f646124ffd2 00007f646124ff01 00007f646124fe7c 00007f646124f132 00007f645f512dfc 00007f645f512236
0114 02:54:23.477811 (+ 3088us) catalog_manager.cc:2076] Acquired catalog manager lock
0114 02:54:23.483734 (+ 5923us) catalog_manager.cc:2122] Created new table and tablet info
0114 02:54:23.691724 (+207990us) catalog_manager.cc:2182] Wrote table and tablets to system table
0114 02:54:23.708387 (+ 16663us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0114 02:54:23.708655 (+ 268us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:23.615793 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:23.626484 (+ 10691us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:23.626769 (+ 285us) write_op.cc:435] Acquired schema lock
0114 02:54:23.630767 (+ 3998us) tablet.cc:659] Decoding operations
0114 02:54:23.676177 (+ 45410us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:23.676245 (+ 68us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:23.676279 (+ 34us) tablet.cc:682] Acquiring locks for 2 operations
0114 02:54:23.676545 (+ 266us) tablet.cc:698] Row locks acquired
0114 02:54:23.676565 (+ 20us) write_op.cc:260] PREPARE: finished
0114 02:54:23.676707 (+ 142us) write_op.cc:270] Start()
0114 02:54:23.676809 (+ 102us) write_op.cc:276] Timestamp: P: 1736823263676620 usec, L: 0
0114 02:54:23.676830 (+ 21us) op_driver.cc:348] REPLICATION: starting
0114 02:54:23.677398 (+ 568us) log.cc:844] Serialized 433 byte log entry
0114 02:54:23.688381 (+ 10983us) op_driver.cc:464] REPLICATION: finished
0114 02:54:23.689451 (+ 1070us) write_op.cc:301] APPLY: starting
0114 02:54:23.689516 (+ 65us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:23.689643 (+ 127us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:23.689757 (+ 114us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:23.690217 (+ 460us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:23.690391 (+ 174us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 02:54:23.690454 (+ 63us) write_op.cc:312] APPLY: finished
0114 02:54:23.690664 (+ 210us) log.cc:844] Serialized 29 byte log entry
0114 02:54:23.691370 (+ 706us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:23.691465 (+ 95us) write_op.cc:454] Released schema lock
0114 02:54:23.691504 (+ 39us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":200188288,"child_traces":[["op",{"apply.queue_time_us":891,"apply.run_cpu_time_us":1759,"apply.run_wall_time_us":2751,"num_ops":2,"prepare.queue_time_us":129205,"raft.queue_time_us":161,"raft.run_cpu_time_us":784,"raft.run_wall_time_us":1597,"replication_time_us":11438}]]}
I20250114 02:54:23.735399 1080 tablet_service.cc:1514] Processing DeleteTablet for tablet e15e6e52d7294aeeb4463b3a76022ec3 with delete_type TABLET_DATA_DELETED (replaced by tablet 6d741bfd00784d2d830be3ebe50dc7c4) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:23.736212 466 catalog_manager.cc:6596] ReplaceTablet: tablet e15e6e52d7294aeeb4463b3a76022ec3 deleted and replaced by tablet 6d741bfd00784d2d830be3ebe50dc7c4
I20250114 02:54:23.736723 466 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:33540 (request call id 230) took 1491 ms. Trace:
I20250114 02:54:23.742282 1899 raft_consensus.cc:491] T ca986f81ed6848a89588366f3f400eec P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:23.743280 1899 raft_consensus.cc:513] T ca986f81ed6848a89588366f3f400eec P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:23.745172 1899 leader_election.cc:290] T ca986f81ed6848a89588366f3f400eec P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:23.748824 1786 consensus_queue.cc:1035] T 0f1ea1608213492c9a216757896ff2a4 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:23.750380 1102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca986f81ed6848a89588366f3f400eec" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:23.751056 1102 raft_consensus.cc:2463] T ca986f81ed6848a89588366f3f400eec P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 0.
I20250114 02:54:23.752437 1820 raft_consensus.cc:491] T 044832851f804208b0a371eb0857dbf3 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:23.752882 1820 raft_consensus.cc:513] T 044832851f804208b0a371eb0857dbf3 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:23.755501 1820 leader_election.cc:290] T 044832851f804208b0a371eb0857dbf3 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:23.757344 753 leader_election.cc:304] T ca986f81ed6848a89588366f3f400eec P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:23.758750 470 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 044832851f804208b0a371eb0857dbf3 (table default.Table_0518d8bed7304c86aff32e05e6f07c0d [id=515a2fe3f1e84e82bccc02ef5cecef1b]). Expected version 1 got 0
I20250114 02:54:23.759639 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "044832851f804208b0a371eb0857dbf3" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:23.726449 465 rpcz_store.cc:276] 0114 02:54:22.150629 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:23.296853 (+1146224us) service_pool.cc:225] Handling call
0114 02:54:23.388173 (+ 91320us) catalog_manager.cc:2076] Acquired catalog manager lock
0114 02:54:23.390078 (+ 1905us) catalog_manager.cc:2122] Created new table and tablet info
0114 02:54:23.536205 (+146127us) catalog_manager.cc:2182] Wrote table and tablets to system table
0114 02:54:23.725613 (+189408us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0114 02:54:23.725934 (+ 321us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:23.489616 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:23.490088 (+ 472us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:23.490119 (+ 31us) write_op.cc:435] Acquired schema lock
0114 02:54:23.490138 (+ 19us) tablet.cc:659] Decoding operations
0114 02:54:23.490478 (+ 340us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:23.490525 (+ 47us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:23.490557 (+ 32us) tablet.cc:682] Acquiring locks for 2 operations
0114 02:54:23.490797 (+ 240us) tablet.cc:698] Row locks acquired
0114 02:54:23.490815 (+ 18us) write_op.cc:260] PREPARE: finished
0114 02:54:23.490936 (+ 121us) write_op.cc:270] Start()
0114 02:54:23.491034 (+ 98us) write_op.cc:276] Timestamp: P: 1736823263490869 usec, L: 0
0114 02:54:23.491054 (+ 20us) op_driver.cc:348] REPLICATION: starting
0114 02:54:23.494319 (+ 3265us) log.cc:844] Serialized 433 byte log entry
0114 02:54:23.530893 (+ 36574us) op_driver.cc:464] REPLICATION: finished
0114 02:54:23.533191 (+ 2298us) write_op.cc:301] APPLY: starting
0114 02:54:23.533251 (+ 60us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:23.533374 (+ 123us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:23.533499 (+ 125us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:23.533952 (+ 453us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:23.534125 (+ 173us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 02:54:23.534182 (+ 57us) write_op.cc:312] APPLY: finished
0114 02:54:23.534408 (+ 226us) log.cc:844] Serialized 29 byte log entry
0114 02:54:23.534565 (+ 157us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:23.534645 (+ 80us) write_op.cc:454] Released schema lock
0114 02:54:23.534683 (+ 38us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:23.555045 (+ 20362us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:23.555121 (+ 76us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2164,"apply.run_cpu_time_us":1694,"apply.run_wall_time_us":1694,"num_ops":2,"prepare.queue_time_us":98118,"prepare.run_cpu_time_us":2443,"prepare.run_wall_time_us":65244,"raft.queue_time_us":1459,"raft.run_cpu_time_us":333,"raft.run_wall_time_us":343,"replication_time_us":39747,"thread_start_us":20724,"threads_started":1}]]}
I20250114 02:54:23.715776 1784 raft_consensus.cc:513] T 0304df55431e42e1a999ed17dc35e6c0 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:23.763406 1784 leader_election.cc:290] T 0304df55431e42e1a999ed17dc35e6c0 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Requested vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:23.763302 813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0304df55431e42e1a999ed17dc35e6c0" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:23.763890 813 raft_consensus.cc:3054] T 0304df55431e42e1a999ed17dc35e6c0 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:23.742097 466 rpcz_store.cc:276] 0114 02:54:22.245060 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:23.379482 (+1134422us) service_pool.cc:225] Handling call
0114 02:54:23.480317 (+100835us) spinlock_profiling.cc:243] Waited 81.9 ms on lock 0x7b54000316d4. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616cad50 00007f646174df2a 00007f64633f2dec 00007f64634eed31 00007f6461270d47 00007f6461270cd2 00007f6461270c01 00007f6461270b7c 00007f646126fe32 00007f645f512dfc 00007f645f512236
0114 02:54:23.736658 (+256341us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:23.555851 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:23.562641 (+ 6790us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:23.562911 (+ 270us) write_op.cc:435] Acquired schema lock
0114 02:54:23.562928 (+ 17us) tablet.cc:659] Decoding operations
0114 02:54:23.567183 (+ 4255us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:23.567483 (+ 300us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:23.567753 (+ 270us) tablet.cc:682] Acquiring locks for 2 operations
0114 02:54:23.569681 (+ 1928us) tablet.cc:698] Row locks acquired
0114 02:54:23.569696 (+ 15us) write_op.cc:260] PREPARE: finished
0114 02:54:23.571364 (+ 1668us) write_op.cc:270] Start()
0114 02:54:23.572855 (+ 1491us) write_op.cc:276] Timestamp: P: 1736823263570482 usec, L: 0
0114 02:54:23.573140 (+ 285us) op_driver.cc:348] REPLICATION: starting
0114 02:54:23.579109 (+ 5969us) log.cc:844] Serialized 625 byte log entry
0114 02:54:23.597865 (+ 18756us) op_driver.cc:464] REPLICATION: finished
0114 02:54:23.603280 (+ 5415us) write_op.cc:301] APPLY: starting
0114 02:54:23.603335 (+ 55us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:23.603447 (+ 112us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:23.603616 (+ 169us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:23.604081 (+ 465us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:23.604332 (+ 251us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:23.604380 (+ 48us) write_op.cc:312] APPLY: finished
0114 02:54:23.604559 (+ 179us) log.cc:844] Serialized 29 byte log entry
0114 02:54:23.605118 (+ 559us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:23.605182 (+ 64us) write_op.cc:454] Released schema lock
0114 02:54:23.605212 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:23.609120 (+ 3908us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:23.609947 (+ 827us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":188340992,"child_traces":[["op",{"apply.queue_time_us":5238,"apply.run_cpu_time_us":1541,"apply.run_wall_time_us":2335,"num_ops":2,"prepare.queue_time_us":72097,"prepare.run_cpu_time_us":3723,"prepare.run_wall_time_us":52025,"raft.queue_time_us":1604,"raft.run_cpu_time_us":687,"raft.run_wall_time_us":15869,"replication_time_us":24258}]]}
I20250114 02:54:23.776444 1899 raft_consensus.cc:2798] T ca986f81ed6848a89588366f3f400eec P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:23.776803 1899 raft_consensus.cc:491] T ca986f81ed6848a89588366f3f400eec P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:23.777086 1899 raft_consensus.cc:3054] T ca986f81ed6848a89588366f3f400eec P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:23.790055 817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "044832851f804208b0a371eb0857dbf3" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:23.791704 960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca986f81ed6848a89588366f3f400eec" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:23.792497 960 raft_consensus.cc:2463] T ca986f81ed6848a89588366f3f400eec P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 0.
I20250114 02:54:23.796622 817 raft_consensus.cc:2388] T 044832851f804208b0a371eb0857dbf3 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 90d6c2c58bb54e0eafb365625ac33871 in current term 1: Already voted for candidate 525198593c8e47afaee54e74267b87e3 in this term.
I20250114 02:54:23.801505 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0304df55431e42e1a999ed17dc35e6c0" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
I20250114 02:54:23.802045 962 raft_consensus.cc:3054] T 0304df55431e42e1a999ed17dc35e6c0 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:23.804056 813 raft_consensus.cc:2463] T 0304df55431e42e1a999ed17dc35e6c0 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1.
I20250114 02:54:23.712921 1206 raft_consensus.cc:357] T 46360f726fad4534a2daf67581346fc1 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:23.806212 1206 raft_consensus.cc:383] T 46360f726fad4534a2daf67581346fc1 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:23.808812 1206 raft_consensus.cc:738] T 46360f726fad4534a2daf67581346fc1 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:23.808945 1042 leader_election.cc:304] T 044832851f804208b0a371eb0857dbf3 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90d6c2c58bb54e0eafb365625ac33871; no voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2
I20250114 02:54:23.810019 1042 leader_election.cc:304] T 0304df55431e42e1a999ed17dc35e6c0 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:23.810752 1841 raft_consensus.cc:3054] T 044832851f804208b0a371eb0857dbf3 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:23.812397 1784 raft_consensus.cc:2798] T 0304df55431e42e1a999ed17dc35e6c0 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:23.812821 1784 raft_consensus.cc:695] T 0304df55431e42e1a999ed17dc35e6c0 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Becoming Leader. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Running, Role: LEADER
I20250114 02:54:23.814929 1899 raft_consensus.cc:513] T ca986f81ed6848a89588366f3f400eec P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:23.822742 1205 tablet_bootstrap.cc:492] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:23.823173 1205 ts_tablet_manager.cc:1397] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.204s user 0.017s sys 0.057s
I20250114 02:54:23.824431 1784 consensus_queue.cc:237] T 0304df55431e42e1a999ed17dc35e6c0 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:23.825472 1205 raft_consensus.cc:357] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:23.826146 1205 raft_consensus.cc:383] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:23.826395 1205 raft_consensus.cc:738] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:23.826963 1205 consensus_queue.cc:260] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:23.713732 462 rpcz_store.cc:276] 0114 02:54:22.036025 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:23.290927 (+1254902us) service_pool.cc:225] Handling call
0114 02:54:23.292188 (+ 1261us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0114 02:54:23.293021 (+ 833us) catalog_manager.cc:3627] Apply alter schema
0114 02:54:23.345656 (+ 52635us) catalog_manager.cc:3659] Acquired catalog manager lock
0114 02:54:23.345957 (+ 301us) catalog_manager.cc:3817] Updating metadata on disk
0114 02:54:23.536046 (+190089us) catalog_manager.cc:3848] Committing alterations to in-memory state
0114 02:54:23.713307 (+177261us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:23.460615 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:23.461083 (+ 468us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:23.461115 (+ 32us) write_op.cc:435] Acquired schema lock
0114 02:54:23.461135 (+ 20us) tablet.cc:659] Decoding operations
0114 02:54:23.461460 (+ 325us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:23.461513 (+ 53us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:23.461546 (+ 33us) tablet.cc:682] Acquiring locks for 1 operations
0114 02:54:23.462365 (+ 819us) tablet.cc:698] Row locks acquired
0114 02:54:23.462388 (+ 23us) write_op.cc:260] PREPARE: finished
0114 02:54:23.462539 (+ 151us) write_op.cc:270] Start()
0114 02:54:23.462644 (+ 105us) write_op.cc:276] Timestamp: P: 1736823263462471 usec, L: 0
0114 02:54:23.462667 (+ 23us) op_driver.cc:348] REPLICATION: starting
0114 02:54:23.487102 (+ 24435us) log.cc:844] Serialized 405 byte log entry
0114 02:54:23.489187 (+ 2085us) spinlock_profiling.cc:243] Waited 22.4 ms on lock 0x7b54000316d4. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616ca6d1 00007f6461741198 00007f6461bad3b1 00007f6461bacb88 00007f6461bb1821 00007f6461bb17d9 00007f6461bb1769 00007f6461bb1731 00007f6461bb095d 000000000059ea04 000000000059e839
0114 02:54:23.525734 (+ 36547us) spinlock_profiling.cc:243] Waited 33.6 ms on lock 0x7b58000d01b0. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616cad50 00007f64616bfa94 00007f64616c49d1 00007f64616c4959 00007f64616c48e9 00007f64616c48b1 00007f64616c3a7d 000000000059ea04 000000000059e839 00007f645a7fc866 00007f645a7ffcc1
0114 02:54:23.530608 (+ 4874us) op_driver.cc:464] REPLICATION: finished
0114 02:54:23.531440 (+ 832us) write_op.cc:301] APPLY: starting
0114 02:54:23.531516 (+ 76us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:23.531634 (+ 118us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:23.531662 (+ 28us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:23.531842 (+ 180us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:23.532016 (+ 174us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:23.532078 (+ 62us) write_op.cc:312] APPLY: finished
0114 02:54:23.532321 (+ 243us) log.cc:844] Serialized 23 byte log entry
0114 02:54:23.532513 (+ 192us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:23.532591 (+ 78us) write_op.cc:454] Released schema lock
0114 02:54:23.532632 (+ 41us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:23.532897 (+ 265us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:23.532931 (+ 34us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":638,"apply.run_cpu_time_us":1421,"apply.run_wall_time_us":1420,"num_ops":1,"prepare.queue_time_us":113190,"prepare.run_cpu_time_us":3296,"prepare.run_wall_time_us":28838,"raft.queue_time_us":15663,"raft.run_cpu_time_us":1893,"raft.run_wall_time_us":35296,"replication_time_us":67814,"spinlock_wait_cycles":128767488}]]}
I20250114 02:54:23.816952 464 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33540 (request call id 229) took 1621 ms. Trace:
I20250114 02:54:23.829104 1205 ts_tablet_manager.cc:1428] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20250114 02:54:23.827955 464 rpcz_store.cc:276] 0114 02:54:22.195615 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:23.375661 (+1180046us) service_pool.cc:225] Handling call
0114 02:54:23.468388 (+ 92727us) spinlock_profiling.cc:243] Waited 91.2 ms on lock 0x7b54000316d4. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616cad50 00007f646174df2a 00007f64633f2dec 00007f64634ec306 00007f6461250044 00007f646124ffd2 00007f646124ff01 00007f646124fe7c 00007f646124f132 00007f645f512dfc 00007f645f512236
0114 02:54:23.471101 (+ 2713us) catalog_manager.cc:2076] Acquired catalog manager lock
0114 02:54:23.492468 (+ 21367us) catalog_manager.cc:2122] Created new table and tablet info
0114 02:54:23.816356 (+323888us) catalog_manager.cc:2182] Wrote table and tablets to system table
0114 02:54:23.816652 (+ 296us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0114 02:54:23.816895 (+ 243us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:23.745912 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:23.746345 (+ 433us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:23.746370 (+ 25us) write_op.cc:435] Acquired schema lock
0114 02:54:23.746385 (+ 15us) tablet.cc:659] Decoding operations
0114 02:54:23.746713 (+ 328us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:23.746760 (+ 47us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:23.746791 (+ 31us) tablet.cc:682] Acquiring locks for 2 operations
0114 02:54:23.747015 (+ 224us) tablet.cc:698] Row locks acquired
0114 02:54:23.747032 (+ 17us) write_op.cc:260] PREPARE: finished
0114 02:54:23.747151 (+ 119us) write_op.cc:270] Start()
0114 02:54:23.747244 (+ 93us) write_op.cc:276] Timestamp: P: 1736823263747059 usec, L: 0
0114 02:54:23.747265 (+ 21us) op_driver.cc:348] REPLICATION: starting
0114 02:54:23.747805 (+ 540us) log.cc:844] Serialized 433 byte log entry
0114 02:54:23.812359 (+ 64554us) op_driver.cc:464] REPLICATION: finished
0114 02:54:23.813963 (+ 1604us) write_op.cc:301] APPLY: starting
0114 02:54:23.814016 (+ 53us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:23.814120 (+ 104us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:23.814219 (+ 99us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:23.814593 (+ 374us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:23.815263 (+ 670us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 02:54:23.815313 (+ 50us) write_op.cc:312] APPLY: finished
0114 02:54:23.815485 (+ 172us) log.cc:844] Serialized 29 byte log entry
0114 02:54:23.816059 (+ 574us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:23.816125 (+ 66us) write_op.cc:454] Released schema lock
0114 02:54:23.816170 (+ 45us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:23.820207 (+ 4037us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:23.820251 (+ 44us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":209821952,"child_traces":[["op",{"apply.queue_time_us":1419,"apply.run_cpu_time_us":1983,"apply.run_wall_time_us":3372,"num_ops":2,"prepare.queue_time_us":251988,"prepare.run_cpu_time_us":2318,"prepare.run_wall_time_us":74280,"raft.queue_time_us":15042,"raft.run_cpu_time_us":843,"raft.run_wall_time_us":1561,"replication_time_us":64969}]]}
I20250114 02:54:23.836328 464 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_44d578b4433242d5b7d8a61f6e4213e7" } new_table_name: "default.Table_f842a61f536940588cd46bedc206f2b3" modify_external_catalogs: true
I20250114 02:54:23.835428 960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca986f81ed6848a89588366f3f400eec" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
I20250114 02:54:23.837743 960 raft_consensus.cc:3054] T ca986f81ed6848a89588366f3f400eec P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:23.838987 462 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_972e696cae0e4f05a434e5375f6767c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:23.840514 462 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_972e696cae0e4f05a434e5375f6767c5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:23.842438 466 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_efc3b170c00f4ce09b2601f7f5ebf3b8" } modify_external_catalogs: true
I20250114 02:54:23.842955 466 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_efc3b170c00f4ce09b2601f7f5ebf3b8" } modify_external_catalogs: true
I20250114 02:54:23.844200 962 raft_consensus.cc:2463] T 0304df55431e42e1a999ed17dc35e6c0 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1.
I20250114 02:54:23.815845 1206 consensus_queue.cc:260] T 46360f726fad4534a2daf67581346fc1 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:23.846565 1205 tablet_bootstrap.cc:492] T 42cee9e8e26147b593ed73fca6ac1183 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:23.852967 1205 tablet_bootstrap.cc:654] T 42cee9e8e26147b593ed73fca6ac1183 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:23.856017 1206 ts_tablet_manager.cc:1428] T 46360f726fad4534a2daf67581346fc1 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.145s user 0.003s sys 0.002s
I20250114 02:54:23.860546 1206 tablet_bootstrap.cc:492] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:23.826951 1841 raft_consensus.cc:2743] T 044832851f804208b0a371eb0857dbf3 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:23.838835 1099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca986f81ed6848a89588366f3f400eec" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:23.866832 1099 raft_consensus.cc:3054] T ca986f81ed6848a89588366f3f400eec P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:23.873869 1099 raft_consensus.cc:2463] T ca986f81ed6848a89588366f3f400eec P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 1.
I20250114 02:54:23.874759 1205 tablet_bootstrap.cc:492] T 42cee9e8e26147b593ed73fca6ac1183 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:23.875149 753 leader_election.cc:304] T ca986f81ed6848a89588366f3f400eec P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters:
W20250114 02:54:23.875702 464 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ed7f575653c848d7b076c17c38c4a244 consensus state has no leader
I20250114 02:54:23.875795 1741 raft_consensus.cc:2798] T ca986f81ed6848a89588366f3f400eec P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:23.876178 464 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:33540 (request call id 232) took 1525 ms. Trace:
I20250114 02:54:23.876488 464 rpcz_store.cc:276] 0114 02:54:22.350733 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:23.835934 (+1485201us) service_pool.cc:225] Handling call
0114 02:54:23.837266 (+ 1332us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0114 02:54:23.849847 (+ 12581us) catalog_manager.cc:3627] Apply alter schema
0114 02:54:23.850441 (+ 594us) catalog_manager.cc:3659] Acquired catalog manager lock
0114 02:54:23.850669 (+ 228us) catalog_manager.cc:3817] Updating metadata on disk
0114 02:54:23.875155 (+ 24486us) catalog_manager.cc:3848] Committing alterations to in-memory state
0114 02:54:23.876118 (+ 963us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:23.851932 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:23.852281 (+ 349us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:23.852304 (+ 23us) write_op.cc:435] Acquired schema lock
0114 02:54:23.852317 (+ 13us) tablet.cc:659] Decoding operations
0114 02:54:23.852549 (+ 232us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:23.852587 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:23.852608 (+ 21us) tablet.cc:682] Acquiring locks for 1 operations
0114 02:54:23.852757 (+ 149us) tablet.cc:698] Row locks acquired
0114 02:54:23.852774 (+ 17us) write_op.cc:260] PREPARE: finished
0114 02:54:23.852920 (+ 146us) write_op.cc:270] Start()
0114 02:54:23.853019 (+ 99us) write_op.cc:276] Timestamp: P: 1736823263852872 usec, L: 0
0114 02:54:23.853033 (+ 14us) op_driver.cc:348] REPLICATION: starting
0114 02:54:23.853487 (+ 454us) log.cc:844] Serialized 405 byte log entry
0114 02:54:23.866623 (+ 13136us) op_driver.cc:464] REPLICATION: finished
0114 02:54:23.873341 (+ 6718us) write_op.cc:301] APPLY: starting
0114 02:54:23.873397 (+ 56us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:23.873498 (+ 101us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:23.873593 (+ 95us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:23.873745 (+ 152us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:23.873866 (+ 121us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:23.873924 (+ 58us) write_op.cc:312] APPLY: finished
0114 02:54:23.874102 (+ 178us) log.cc:844] Serialized 23 byte log entry
0114 02:54:23.874259 (+ 157us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:23.874318 (+ 59us) write_op.cc:454] Released schema lock
0114 02:54:23.874346 (+ 28us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":6562,"apply.run_cpu_time_us":1180,"apply.run_wall_time_us":1180,"num_ops":1,"prepare.queue_time_us":102,"raft.queue_time_us":86,"raft.run_cpu_time_us":717,"raft.run_wall_time_us":718,"replication_time_us":13503}]]}
I20250114 02:54:23.875123 1205 ts_tablet_manager.cc:1397] T 42cee9e8e26147b593ed73fca6ac1183 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.029s user 0.007s sys 0.005s
I20250114 02:54:23.864754 1919 raft_consensus.cc:491] T f2b6812e8c1d44529d0284be490df648 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:23.880851 464 master_service.cc:946] ReplaceTablet: received request to replace tablet 0f1ea1608213492c9a216757896ff2a4 from {username='slave'} at 127.0.0.1:33540
I20250114 02:54:23.880589 1919 raft_consensus.cc:513] T f2b6812e8c1d44529d0284be490df648 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:23.883574 813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2b6812e8c1d44529d0284be490df648" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:23.887487 1205 raft_consensus.cc:357] T 42cee9e8e26147b593ed73fca6ac1183 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:23.888192 1205 raft_consensus.cc:383] T 42cee9e8e26147b593ed73fca6ac1183 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:23.888477 1205 raft_consensus.cc:738] T 42cee9e8e26147b593ed73fca6ac1183 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:23.831383 1899 leader_election.cc:290] T ca986f81ed6848a89588366f3f400eec P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Requested vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:23.892632 1205 consensus_queue.cc:260] T 42cee9e8e26147b593ed73fca6ac1183 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:23.899641 960 raft_consensus.cc:2463] T ca986f81ed6848a89588366f3f400eec P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 1.
I20250114 02:54:23.905064 1741 raft_consensus.cc:695] T ca986f81ed6848a89588366f3f400eec P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 LEADER]: Becoming Leader. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Running, Role: LEADER
I20250114 02:54:23.906044 1741 consensus_queue.cc:237] T ca986f81ed6848a89588366f3f400eec P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:23.910434 1595 ts_tablet_manager.cc:1918] T 66bfa92db9874750b3c340e31a06c412 P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:23.910825 1595 log.cc:1198] T 66bfa92db9874750b3c340e31a06c412 P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/66bfa92db9874750b3c340e31a06c412
I20250114 02:54:23.911518 1595 ts_tablet_manager.cc:1939] T 66bfa92db9874750b3c340e31a06c412 P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
I20250114 02:54:23.916318 1919 leader_election.cc:290] T f2b6812e8c1d44529d0284be490df648 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:23.917874 1207 tablet_bootstrap.cc:492] T 46360f726fad4534a2daf67581346fc1 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:23.918427 1207 ts_tablet_manager.cc:1397] T 46360f726fad4534a2daf67581346fc1 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.252s user 0.011s sys 0.000s
I20250114 02:54:23.920521 1207 raft_consensus.cc:357] T 46360f726fad4534a2daf67581346fc1 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:23.903584 1205 ts_tablet_manager.cc:1428] T 42cee9e8e26147b593ed73fca6ac1183 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.023s user 0.002s sys 0.003s
I20250114 02:54:23.909551 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2b6812e8c1d44529d0284be490df648" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:23.921967 962 raft_consensus.cc:2388] T f2b6812e8c1d44529d0284be490df648 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 90d6c2c58bb54e0eafb365625ac33871 in current term 1: Already voted for candidate 819f9e3412e54d44abb11d8b3ca877c2 in this term.
I20250114 02:54:23.923418 1205 tablet_bootstrap.cc:492] T ddfbab181b1f474192c664cd0316037f P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:23.934494 1594 ts_tablet_manager.cc:1918] T 26328f8e32bc49ed9b224f0e08b9f949 P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250114 02:54:23.934866 1594 log.cc:1198] T 26328f8e32bc49ed9b224f0e08b9f949 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/26328f8e32bc49ed9b224f0e08b9f949
I20250114 02:54:23.935529 1594 ts_tablet_manager.cc:1939] T 26328f8e32bc49ed9b224f0e08b9f949 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:23.937757 1206 tablet_bootstrap.cc:654] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:23.939713 1205 tablet_bootstrap.cc:654] T ddfbab181b1f474192c664cd0316037f P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:23.949067 1205 tablet_bootstrap.cc:492] T ddfbab181b1f474192c664cd0316037f P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:23.949461 1205 ts_tablet_manager.cc:1397] T ddfbab181b1f474192c664cd0316037f P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.026s user 0.010s sys 0.000s
I20250114 02:54:23.952136 1205 raft_consensus.cc:357] T ddfbab181b1f474192c664cd0316037f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:23.952787 1205 raft_consensus.cc:383] T ddfbab181b1f474192c664cd0316037f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:23.953052 1205 raft_consensus.cc:738] T ddfbab181b1f474192c664cd0316037f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:23.954547 1702 ts_tablet_manager.cc:1918] T bc46815b4803418c95ea9c039a4fae88 P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:23.954934 1702 log.cc:1198] T bc46815b4803418c95ea9c039a4fae88 P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/bc46815b4803418c95ea9c039a4fae88
I20250114 02:54:23.955963 1702 ts_tablet_manager.cc:1939] T bc46815b4803418c95ea9c039a4fae88 P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:23.922880 1040 leader_election.cc:304] T f2b6812e8c1d44529d0284be490df648 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90d6c2c58bb54e0eafb365625ac33871; no voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2
I20250114 02:54:23.956607 1919 raft_consensus.cc:3054] T f2b6812e8c1d44529d0284be490df648 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:23.957284 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet 66bfa92db9874750b3c340e31a06c412 (table default.Table_b0cd93e5a498450e84fb0f1ba4be4505 [id=c70751b59bfc4c6fa60d0e8bb40d5f36]) successfully deleted
I20250114 02:54:23.957942 1207 raft_consensus.cc:383] T 46360f726fad4534a2daf67581346fc1 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:23.958292 1207 raft_consensus.cc:738] T 46360f726fad4534a2daf67581346fc1 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:23.959136 1207 consensus_queue.cc:260] T 46360f726fad4534a2daf67581346fc1 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:23.961279 1207 ts_tablet_manager.cc:1428] T 46360f726fad4534a2daf67581346fc1 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.043s user 0.005s sys 0.000s
I20250114 02:54:23.963932 462 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33540 (request call id 233) took 1509 ms. Trace:
W20250114 02:54:23.969703 458 catalog_manager.cc:4852] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet e15e6e52d7294aeeb4463b3a76022ec3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e15e6e52d7294aeeb4463b3a76022ec3 already in progress: creating tablet
I20250114 02:54:23.975512 1594 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 185) took 1505 ms. Trace:
I20250114 02:54:23.975859 1594 rpcz_store.cc:276] 0114 02:54:22.470494 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:22.470618 (+ 124us) service_pool.cc:225] Handling call
0114 02:54:23.550673 (+1080055us) ts_tablet_manager.cc:1145] Deleting tablet 26328f8e32bc49ed9b224f0e08b9f949
0114 02:54:23.592309 (+ 41636us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:23.731127 (+138818us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:23.934279 (+203152us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:23.975453 (+ 41174us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1079458}
I20250114 02:54:23.976493 1207 tablet_bootstrap.cc:492] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:23.976994 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet 26328f8e32bc49ed9b224f0e08b9f949 (table default.Table_f851d0fbfd4e4095912dbe3a96fd7142 [id=a01ec721fefc41d6bc4009ca0c327fab]) successfully deleted
I20250114 02:54:23.964385 462 rpcz_store.cc:276] 0114 02:54:22.454501 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:23.838551 (+1384050us) service_pool.cc:225] Handling call
0114 02:54:23.841766 (+ 3215us) catalog_manager.cc:2076] Acquired catalog manager lock
0114 02:54:23.857001 (+ 15235us) catalog_manager.cc:2122] Created new table and tablet info
0114 02:54:23.948359 (+ 91358us) catalog_manager.cc:2182] Wrote table and tablets to system table
0114 02:54:23.963520 (+ 15161us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0114 02:54:23.963856 (+ 336us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:23.883972 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:23.884362 (+ 390us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:23.884384 (+ 22us) write_op.cc:435] Acquired schema lock
0114 02:54:23.884397 (+ 13us) tablet.cc:659] Decoding operations
0114 02:54:23.888085 (+ 3688us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:23.888126 (+ 41us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:23.888146 (+ 20us) tablet.cc:682] Acquiring locks for 2 operations
0114 02:54:23.888365 (+ 219us) tablet.cc:698] Row locks acquired
0114 02:54:23.888379 (+ 14us) write_op.cc:260] PREPARE: finished
0114 02:54:23.888488 (+ 109us) write_op.cc:270] Start()
0114 02:54:23.888571 (+ 83us) write_op.cc:276] Timestamp: P: 1736823263888436 usec, L: 0
0114 02:54:23.888587 (+ 16us) op_driver.cc:348] REPLICATION: starting
0114 02:54:23.889021 (+ 434us) log.cc:844] Serialized 433 byte log entry
0114 02:54:23.943655 (+ 54634us) op_driver.cc:464] REPLICATION: finished
0114 02:54:23.944265 (+ 610us) write_op.cc:301] APPLY: starting
0114 02:54:23.944337 (+ 72us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:23.944471 (+ 134us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:23.944593 (+ 122us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:23.945072 (+ 479us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:23.945201 (+ 129us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 02:54:23.945238 (+ 37us) write_op.cc:312] APPLY: finished
0114 02:54:23.945397 (+ 159us) log.cc:844] Serialized 29 byte log entry
0114 02:54:23.945532 (+ 135us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:23.945597 (+ 65us) write_op.cc:454] Released schema lock
0114 02:54:23.945639 (+ 42us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:23.945855 (+ 216us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:23.945880 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":412,"apply.run_cpu_time_us":1580,"apply.run_wall_time_us":1580,"num_ops":2,"prepare.queue_time_us":25674,"prepare.run_cpu_time_us":1976,"prepare.run_wall_time_us":29352,"raft.queue_time_us":4244,"raft.run_cpu_time_us":666,"raft.run_wall_time_us":666,"replication_time_us":54980}]]}
I20250114 02:54:23.978245 1594 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 189) took 1227 ms. Trace:
I20250114 02:54:23.953644 1205 consensus_queue.cc:260] T ddfbab181b1f474192c664cd0316037f P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:23.978961 1884 raft_consensus.cc:491] T ddfbab181b1f474192c664cd0316037f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:23.980020 462 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_7058838b6bcb427786778096cf6f52b3" } modify_external_catalogs: true
I20250114 02:54:23.979900 1884 raft_consensus.cc:513] T ddfbab181b1f474192c664cd0316037f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:23.980427 1205 ts_tablet_manager.cc:1428] T ddfbab181b1f474192c664cd0316037f P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.031s user 0.007s sys 0.000s
W20250114 02:54:23.980809 456 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet ed7f575653c848d7b076c17c38c4a244 (table default.Table_f842a61f536940588cd46bedc206f2b3 [id=954669573a3c4a97a488805e05b0b220], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet ed7f575653c848d7b076c17c38c4a244 consensus state has no leader
I20250114 02:54:23.981293 1205 tablet_bootstrap.cc:492] T ce7d6088fc884a248918d2348adcaccf P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:23.982311 462 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_7058838b6bcb427786778096cf6f52b3" } modify_external_catalogs: true
I20250114 02:54:23.982851 1702 tablet_replica.cc:331] T 66bfa92db9874750b3c340e31a06c412 P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:23.987272 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet bc46815b4803418c95ea9c039a4fae88 (table default.Table_6a4fe39298514187b65def5841d3a992 [id=5c5a7974ad2f4e58ba5c47a6eeaaac62]) successfully deleted
W20250114 02:54:23.987864 456 catalog_manager.cc:4839] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet 266da813b4514329be0ad97d07b7114f because the tablet was not found. No further retry: Not found: Tablet not found: 266da813b4514329be0ad97d07b7114f
I20250114 02:54:23.988265 1099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddfbab181b1f474192c664cd0316037f" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
W20250114 02:54:23.988554 456 catalog_manager.cc:4839] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet 9812754c610548d1b43084bbc94fc3e4 because the tablet was not found. No further retry: Not found: Tablet not found: 9812754c610548d1b43084bbc94fc3e4
I20250114 02:54:23.998875 1594 rpcz_store.cc:276] 0114 02:54:22.750444 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:22.750605 (+ 161us) service_pool.cc:225] Handling call
0114 02:54:23.977912 (+1227307us) ts_tablet_manager.cc:1145] Deleting tablet c86562688d5b4fffb71cbdf7be0afc37
0114 02:54:23.977952 (+ 40us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:23.978192 (+ 240us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1197447}
W20250114 02:54:24.000118 458 catalog_manager.cc:4852] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet c86562688d5b4fffb71cbdf7be0afc37 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c86562688d5b4fffb71cbdf7be0afc37 already in progress: creating tablet
I20250114 02:54:23.994431 813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddfbab181b1f474192c664cd0316037f" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:23.998088 1206 tablet_bootstrap.cc:492] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:24.001288 1919 raft_consensus.cc:2743] T f2b6812e8c1d44529d0284be490df648 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:24.001785 1206 ts_tablet_manager.cc:1397] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.141s user 0.009s sys 0.000s
I20250114 02:54:24.009167 1820 raft_consensus.cc:491] T 46360f726fad4534a2daf67581346fc1 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:24.009596 1820 raft_consensus.cc:513] T 46360f726fad4534a2daf67581346fc1 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:24.011526 1820 leader_election.cc:290] T 46360f726fad4534a2daf67581346fc1 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:24.013726 814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46360f726fad4534a2daf67581346fc1" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:24.014322 814 raft_consensus.cc:2463] T 46360f726fad4534a2daf67581346fc1 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 0.
I20250114 02:54:24.021085 1042 leader_election.cc:304] T 46360f726fad4534a2daf67581346fc1 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:23.983161 1884 leader_election.cc:290] T ddfbab181b1f474192c664cd0316037f P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:23.960165 1786 raft_consensus.cc:491] T 42cee9e8e26147b593ed73fca6ac1183 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:24.021806 1702 raft_consensus.cc:2238] T 66bfa92db9874750b3c340e31a06c412 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:24.022258 1702 raft_consensus.cc:2267] T 66bfa92db9874750b3c340e31a06c412 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:24.022054 1786 raft_consensus.cc:513] T 42cee9e8e26147b593ed73fca6ac1183 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:24.023944 1702 ts_tablet_manager.cc:1905] T 66bfa92db9874750b3c340e31a06c412 P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:23.993496 1205 tablet_bootstrap.cc:654] T ce7d6088fc884a248918d2348adcaccf P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:24.004007 1206 raft_consensus.cc:357] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:24.024729 1206 raft_consensus.cc:383] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:24.025012 1206 raft_consensus.cc:738] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:24.025652 1206 consensus_queue.cc:260] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:23.993589 1880 raft_consensus.cc:491] T 46360f726fad4534a2daf67581346fc1 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:24.026470 1880 raft_consensus.cc:513] T 46360f726fad4534a2daf67581346fc1 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:24.028283 1880 leader_election.cc:290] T 46360f726fad4534a2daf67581346fc1 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:24.029045 1786 leader_election.cc:290] T 42cee9e8e26147b593ed73fca6ac1183 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:24.040400 1919 raft_consensus.cc:2798] T 46360f726fad4534a2daf67581346fc1 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:24.041304 1919 raft_consensus.cc:491] T 46360f726fad4534a2daf67581346fc1 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:24.042094 1594 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 197) took 1005 ms. Trace:
I20250114 02:54:24.042584 1919 raft_consensus.cc:3054] T 46360f726fad4534a2daf67581346fc1 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:24.042404 1594 rpcz_store.cc:276] 0114 02:54:23.036836 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:23.036969 (+ 133us) service_pool.cc:225] Handling call
0114 02:54:24.041666 (+1004697us) ts_tablet_manager.cc:1145] Deleting tablet 97f4e061d3b94ee9982410b7ce135453
0114 02:54:24.041725 (+ 59us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:24.042018 (+ 293us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1004104}
I20250114 02:54:24.043682 1594 tablet_replica.cc:331] T 66bfa92db9874750b3c340e31a06c412 P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:24.044514 1594 raft_consensus.cc:2238] T 66bfa92db9874750b3c340e31a06c412 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:24.045082 1594 raft_consensus.cc:2267] T 66bfa92db9874750b3c340e31a06c412 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:24.046849 1594 ts_tablet_manager.cc:1905] T 66bfa92db9874750b3c340e31a06c412 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:24.050858 1207 tablet_bootstrap.cc:654] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
W20250114 02:54:24.056830 458 catalog_manager.cc:4852] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet 97f4e061d3b94ee9982410b7ce135453 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 97f4e061d3b94ee9982410b7ce135453 already in progress: creating tablet
I20250114 02:54:24.029593 1206 ts_tablet_manager.cc:1428] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.028s user 0.005s sys 0.000s
I20250114 02:54:24.062906 1206 tablet_bootstrap.cc:492] T 84738a6c78b24c1fb64013048763b18f P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:24.067114 793 tablet_service.cc:1467] Processing CreateTablet for tablet e06daadd346d45969ad5efcf03bc0870 (DEFAULT_TABLE table=default.Table_1a9735a294c2436e84ec9db5e0aa44e4 [id=6922b5f98cc14e1d8a7e745211aa7b45]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:24.067139 789 tablet_service.cc:1467] Processing CreateTablet for tablet ed109b49f20642f988161a3df7872419 (DEFAULT_TABLE table=default.Table_529c38ed585343acb0f42ac60f171b59 [id=af3fec2e3d0243bfbed7aa314a9732a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:24.068113 793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e06daadd346d45969ad5efcf03bc0870. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:24.065095 1080 tablet_service.cc:1467] Processing CreateTablet for tablet e06daadd346d45969ad5efcf03bc0870 (DEFAULT_TABLE table=default.Table_1a9735a294c2436e84ec9db5e0aa44e4 [id=6922b5f98cc14e1d8a7e745211aa7b45]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:24.070617 1080 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e06daadd346d45969ad5efcf03bc0870. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:24.072286 789 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed109b49f20642f988161a3df7872419. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:24.072818 1205 tablet_bootstrap.cc:492] T ce7d6088fc884a248918d2348adcaccf P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:24.073179 1205 ts_tablet_manager.cc:1397] T ce7d6088fc884a248918d2348adcaccf P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.092s user 0.013s sys 0.003s
I20250114 02:54:24.075940 1205 raft_consensus.cc:357] T ce7d6088fc884a248918d2348adcaccf P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:24.076624 1205 raft_consensus.cc:383] T ce7d6088fc884a248918d2348adcaccf P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:24.076851 1205 raft_consensus.cc:738] T ce7d6088fc884a248918d2348adcaccf P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:24.077360 1205 consensus_queue.cc:260] T ce7d6088fc884a248918d2348adcaccf P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:24.077860 1085 tablet_service.cc:1467] Processing CreateTablet for tablet ed109b49f20642f988161a3df7872419 (DEFAULT_TABLE table=default.Table_529c38ed585343acb0f42ac60f171b59 [id=af3fec2e3d0243bfbed7aa314a9732a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:24.078840 1085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed109b49f20642f988161a3df7872419. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:24.079622 1205 ts_tablet_manager.cc:1428] T ce7d6088fc884a248918d2348adcaccf P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250114 02:54:24.080392 1205 tablet_bootstrap.cc:492] T 5ac7708550954a209e2a8d217c3b39bd P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:24.094383 467 catalog_manager.cc:5526] T 6448eea21ab2422c97da2494658ded3f P 90d6c2c58bb54e0eafb365625ac33871 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67). New cstate: current_term: 1 leader_uuid: "90d6c2c58bb54e0eafb365625ac33871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } }
I20250114 02:54:24.095580 467 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet 6448eea21ab2422c97da2494658ded3f (table default.Table_0931c459ae794d2091fcc0f51d031548 [id=e279c31bd63b40b9ade03a3b928bf153]). Expected version 2 got 0
I20250114 02:54:24.096967 1919 raft_consensus.cc:513] T 46360f726fad4534a2daf67581346fc1 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:24.099298 1207 tablet_bootstrap.cc:492] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:24.105118 1207 ts_tablet_manager.cc:1397] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.129s user 0.011s sys 0.000s
I20250114 02:54:24.106262 1206 tablet_bootstrap.cc:654] T 84738a6c78b24c1fb64013048763b18f P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:24.107434 1207 raft_consensus.cc:357] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:24.108114 1207 raft_consensus.cc:383] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:24.108379 1207 raft_consensus.cc:738] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:24.110800 814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46360f726fad4534a2daf67581346fc1" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:24.111516 814 raft_consensus.cc:3054] T 46360f726fad4534a2daf67581346fc1 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:24.117642 814 raft_consensus.cc:2463] T 46360f726fad4534a2daf67581346fc1 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1.
I20250114 02:54:24.118656 1042 leader_election.cc:304] T 46360f726fad4534a2daf67581346fc1 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:24.119324 1820 raft_consensus.cc:2798] T 46360f726fad4534a2daf67581346fc1 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:24.119766 1820 raft_consensus.cc:695] T 46360f726fad4534a2daf67581346fc1 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Becoming Leader. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Running, Role: LEADER
I20250114 02:54:24.119863 1919 leader_election.cc:290] T 46360f726fad4534a2daf67581346fc1 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Requested vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:24.120795 1820 consensus_queue.cc:237] T 46360f726fad4534a2daf67581346fc1 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:24.122570 467 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet 55b68586ef8b45439a7dd8dcd2efcd79 (table default.Table_66f2263162f64250a0b49073abe4bd4c [id=c73ede1a5f664d8eb204fb9a012f8f7a]). Expected version 1 got 0
I20250114 02:54:24.124387 467 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet 0304df55431e42e1a999ed17dc35e6c0 (table default.Table_4c3843233dda407395ab9652d5870bc6 [id=0d173fdb502f482fb86b6d3e5fdc66f1]). Expected version 1 got 0
I20250114 02:54:24.124852 462 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet 84738a6c78b24c1fb64013048763b18f
I20250114 02:54:24.124640 1207 consensus_queue.cc:260] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:24.127331 1207 ts_tablet_manager.cc:1428] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.022s user 0.005s sys 0.000s
I20250114 02:54:24.131111 1081 tablet_service.cc:1514] Processing DeleteTablet for tablet 84738a6c78b24c1fb64013048763b18f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:23 UTC) from {username='slave'} at 127.0.0.1:53876
W20250114 02:54:24.133905 458 catalog_manager.cc:4852] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 84738a6c78b24c1fb64013048763b18f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 84738a6c78b24c1fb64013048763b18f already in progress: creating tablet
I20250114 02:54:24.134685 1205 tablet_bootstrap.cc:654] T 5ac7708550954a209e2a8d217c3b39bd P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:24.128142 462 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_47299fab0314433585117eb56b89c0b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:24.135831 789 tablet_service.cc:1514] Processing DeleteTablet for tablet 84738a6c78b24c1fb64013048763b18f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:23 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:24.149431 1206 tablet_bootstrap.cc:492] T 84738a6c78b24c1fb64013048763b18f P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:24.149834 1206 ts_tablet_manager.cc:1397] T 84738a6c78b24c1fb64013048763b18f P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.088s user 0.009s sys 0.000s
I20250114 02:54:24.128374 1207 tablet_bootstrap.cc:492] T 42cee9e8e26147b593ed73fca6ac1183 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:24.151916 1206 raft_consensus.cc:357] T 84738a6c78b24c1fb64013048763b18f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:24.153714 1206 raft_consensus.cc:383] T 84738a6c78b24c1fb64013048763b18f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:24.153941 1206 raft_consensus.cc:738] T 84738a6c78b24c1fb64013048763b18f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:24.155369 1702 ts_tablet_manager.cc:1918] T 66bfa92db9874750b3c340e31a06c412 P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:24.155736 1702 log.cc:1198] T 66bfa92db9874750b3c340e31a06c412 P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/66bfa92db9874750b3c340e31a06c412
I20250114 02:54:24.160550 1206 consensus_queue.cc:260] T 84738a6c78b24c1fb64013048763b18f P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:24.162748 1702 ts_tablet_manager.cc:1939] T 66bfa92db9874750b3c340e31a06c412 P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:24.168637 1207 tablet_bootstrap.cc:654] T 42cee9e8e26147b593ed73fca6ac1183 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:24.170372 1206 ts_tablet_manager.cc:1428] T 84738a6c78b24c1fb64013048763b18f P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.020s user 0.003s sys 0.002s
I20250114 02:54:24.171252 1206 tablet_bootstrap.cc:492] T 42cee9e8e26147b593ed73fca6ac1183 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:24.188407 1594 ts_tablet_manager.cc:1918] T 66bfa92db9874750b3c340e31a06c412 P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:24.188977 1594 log.cc:1198] T 66bfa92db9874750b3c340e31a06c412 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/66bfa92db9874750b3c340e31a06c412
I20250114 02:54:24.197219 1206 tablet_bootstrap.cc:654] T 42cee9e8e26147b593ed73fca6ac1183 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:24.200135 1594 ts_tablet_manager.cc:1939] T 66bfa92db9874750b3c340e31a06c412 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:24.203940 1594 tablet_replica.cc:331] T 84738a6c78b24c1fb64013048763b18f P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:24.215852 1594 raft_consensus.cc:2238] T 84738a6c78b24c1fb64013048763b18f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:24.218305 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet 66bfa92db9874750b3c340e31a06c412 (table default.Table_b0cd93e5a498450e84fb0f1ba4be4505 [id=c70751b59bfc4c6fa60d0e8bb40d5f36]) successfully deleted
I20250114 02:54:24.218678 1081 tablet_service.cc:1467] Processing CreateTablet for tablet 2e17f619ed2745078541cc1c924aff96 (DEFAULT_TABLE table=default.Table_1142a1a45945449ea16f5501c3adeb8b [id=b124b43245d14d7daebf8d33882c9124]), partition=RANGE (key) PARTITION UNBOUNDED
W20250114 02:54:24.219293 458 catalog_manager.cc:4839] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet ba215eb8422d4f879dd0bf5996543962 because the tablet was not found. No further retry: Not found: Tablet not found: ba215eb8422d4f879dd0bf5996543962
W20250114 02:54:24.219949 458 catalog_manager.cc:4839] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet 266da813b4514329be0ad97d07b7114f because the tablet was not found. No further retry: Not found: Tablet not found: 266da813b4514329be0ad97d07b7114f
I20250114 02:54:24.220187 1081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e17f619ed2745078541cc1c924aff96. 1 dirs total, 0 dirs full, 0 dirs failed
W20250114 02:54:24.220597 458 catalog_manager.cc:4852] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet e15e6e52d7294aeeb4463b3a76022ec3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e15e6e52d7294aeeb4463b3a76022ec3 already in progress: creating tablet
I20250114 02:54:24.221930 1594 raft_consensus.cc:2267] T 84738a6c78b24c1fb64013048763b18f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:24.226255 1594 ts_tablet_manager.cc:1905] T 84738a6c78b24c1fb64013048763b18f P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:24.236316 1206 tablet_bootstrap.cc:492] T 42cee9e8e26147b593ed73fca6ac1183 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:24.236784 1206 ts_tablet_manager.cc:1397] T 42cee9e8e26147b593ed73fca6ac1183 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.066s user 0.010s sys 0.000s
I20250114 02:54:24.238963 1206 raft_consensus.cc:357] T 42cee9e8e26147b593ed73fca6ac1183 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:24.239601 1206 raft_consensus.cc:383] T 42cee9e8e26147b593ed73fca6ac1183 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:24.239908 1206 raft_consensus.cc:738] T 42cee9e8e26147b593ed73fca6ac1183 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:24.249220 1206 consensus_queue.cc:260] T 42cee9e8e26147b593ed73fca6ac1183 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
W20250114 02:54:24.136137 462 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_47299fab0314433585117eb56b89c0b6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:24.260015 789 tablet_service.cc:1467] Processing CreateTablet for tablet 2e17f619ed2745078541cc1c924aff96 (DEFAULT_TABLE table=default.Table_1142a1a45945449ea16f5501c3adeb8b [id=b124b43245d14d7daebf8d33882c9124]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:24.261183 789 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e17f619ed2745078541cc1c924aff96. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:24.261760 1206 ts_tablet_manager.cc:1428] T 42cee9e8e26147b593ed73fca6ac1183 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.025s user 0.005s sys 0.000s
I20250114 02:54:24.262673 1206 tablet_bootstrap.cc:492] T 5ac7708550954a209e2a8d217c3b39bd P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:24.272526 1205 tablet_bootstrap.cc:492] T 5ac7708550954a209e2a8d217c3b39bd P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:24.273154 1205 ts_tablet_manager.cc:1397] T 5ac7708550954a209e2a8d217c3b39bd P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.193s user 0.019s sys 0.013s
I20250114 02:54:24.275871 1205 raft_consensus.cc:357] T 5ac7708550954a209e2a8d217c3b39bd P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:24.276651 1205 raft_consensus.cc:383] T 5ac7708550954a209e2a8d217c3b39bd P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:24.276916 1205 raft_consensus.cc:738] T 5ac7708550954a209e2a8d217c3b39bd P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:24.285609 788 tablet_service.cc:1467] Processing CreateTablet for tablet 797d6cdf36704c98aa106aac87a945ca (DEFAULT_TABLE table=default.Table_5a9480c6474848d6b400e7dd274d15b0 [id=a17ad30ef32f4cbcb22b5a7d1c7096ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:24.285996 1207 tablet_bootstrap.cc:492] T 42cee9e8e26147b593ed73fca6ac1183 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:24.286397 1207 ts_tablet_manager.cc:1397] T 42cee9e8e26147b593ed73fca6ac1183 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.158s user 0.019s sys 0.007s
I20250114 02:54:24.286957 788 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 797d6cdf36704c98aa106aac87a945ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:24.291297 1206 tablet_bootstrap.cc:654] T 5ac7708550954a209e2a8d217c3b39bd P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:24.312743 1594 ts_tablet_manager.cc:1918] T 84738a6c78b24c1fb64013048763b18f P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:24.313133 1594 log.cc:1198] T 84738a6c78b24c1fb64013048763b18f P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/84738a6c78b24c1fb64013048763b18f
I20250114 02:54:24.313764 1594 ts_tablet_manager.cc:1939] T 84738a6c78b24c1fb64013048763b18f P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:24.319380 464 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet 0f1ea1608213492c9a216757896ff2a4
I20250114 02:54:24.320955 464 catalog_manager.cc:6596] ReplaceTablet: tablet 0f1ea1608213492c9a216757896ff2a4 deleted and replaced by tablet 550e84ac4d1b4ef58dfdea6e552ee940
I20250114 02:54:24.322486 1206 tablet_bootstrap.cc:492] T 5ac7708550954a209e2a8d217c3b39bd P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:24.322904 1206 ts_tablet_manager.cc:1397] T 5ac7708550954a209e2a8d217c3b39bd P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.060s user 0.008s sys 0.000s
I20250114 02:54:24.324671 464 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:33540 (request call id 235) took 1267 ms. Trace:
I20250114 02:54:24.325289 1206 raft_consensus.cc:357] T 5ac7708550954a209e2a8d217c3b39bd P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:24.326103 1206 raft_consensus.cc:383] T 5ac7708550954a209e2a8d217c3b39bd P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:24.307430 1207 raft_consensus.cc:357] T 42cee9e8e26147b593ed73fca6ac1183 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:24.333667 469 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 55b68586ef8b45439a7dd8dcd2efcd79 (table default.Table_66f2263162f64250a0b49073abe4bd4c [id=c73ede1a5f664d8eb204fb9a012f8f7a]). Expected version 1 got 0
I20250114 02:54:24.336045 469 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 0304df55431e42e1a999ed17dc35e6c0 (table default.Table_4c3843233dda407395ab9652d5870bc6 [id=0d173fdb502f482fb86b6d3e5fdc66f1]). Expected version 1 got 0
I20250114 02:54:24.326449 1206 raft_consensus.cc:738] T 5ac7708550954a209e2a8d217c3b39bd P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:24.340312 1206 consensus_queue.cc:260] T 5ac7708550954a209e2a8d217c3b39bd P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:24.327523 1207 raft_consensus.cc:383] T 42cee9e8e26147b593ed73fca6ac1183 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:24.344502 1207 raft_consensus.cc:738] T 42cee9e8e26147b593ed73fca6ac1183 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:24.349303 1081 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f1ea1608213492c9a216757896ff2a4 with delete_type TABLET_DATA_DELETED (replaced by tablet 550e84ac4d1b4ef58dfdea6e552ee940) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:24.350129 1595 tablet_replica.cc:331] T 0f1ea1608213492c9a216757896ff2a4 P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:24.351141 1595 raft_consensus.cc:2238] T 0f1ea1608213492c9a216757896ff2a4 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:24.351752 1595 raft_consensus.cc:2267] T 0f1ea1608213492c9a216757896ff2a4 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:24.354423 1595 ts_tablet_manager.cc:1905] T 0f1ea1608213492c9a216757896ff2a4 P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:24.367718 1206 ts_tablet_manager.cc:1428] T 5ac7708550954a209e2a8d217c3b39bd P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.045s user 0.008s sys 0.000s
I20250114 02:54:24.348933 1207 consensus_queue.cc:260] T 42cee9e8e26147b593ed73fca6ac1183 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:24.368970 1206 tablet_bootstrap.cc:492] T ce7d6088fc884a248918d2348adcaccf P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:24.325523 464 rpcz_store.cc:276] 0114 02:54:23.057811 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:23.880730 (+822919us) service_pool.cc:225] Handling call
0114 02:54:23.883431 (+ 2701us) spinlock_profiling.cc:243] Waited 1.26 ms on lock 0x7b54000316d4. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616cad50 00007f646174df2a 00007f64633f2dec 00007f64634eed31 00007f6461270d47 00007f6461270cd2 00007f6461270c01 00007f6461270b7c 00007f646126fe32 00007f645f512dfc 00007f645f512236
0114 02:54:24.324603 (+441172us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:24.180598 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:24.180911 (+ 313us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:24.180932 (+ 21us) write_op.cc:435] Acquired schema lock
0114 02:54:24.180944 (+ 12us) tablet.cc:659] Decoding operations
0114 02:54:24.181211 (+ 267us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:24.181241 (+ 30us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:24.181260 (+ 19us) tablet.cc:682] Acquiring locks for 2 operations
0114 02:54:24.181432 (+ 172us) tablet.cc:698] Row locks acquired
0114 02:54:24.181445 (+ 13us) write_op.cc:260] PREPARE: finished
0114 02:54:24.181540 (+ 95us) write_op.cc:270] Start()
0114 02:54:24.181613 (+ 73us) write_op.cc:276] Timestamp: P: 1736823264181460 usec, L: 0
0114 02:54:24.181626 (+ 13us) op_driver.cc:348] REPLICATION: starting
0114 02:54:24.182038 (+ 412us) log.cc:844] Serialized 625 byte log entry
0114 02:54:24.309447 (+127409us) op_driver.cc:464] REPLICATION: finished
0114 02:54:24.316982 (+ 7535us) write_op.cc:301] APPLY: starting
0114 02:54:24.317052 (+ 70us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:24.317186 (+ 134us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:24.317307 (+ 121us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:24.317710 (+ 403us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:24.317965 (+ 255us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:24.318027 (+ 62us) write_op.cc:312] APPLY: finished
0114 02:54:24.318244 (+ 217us) log.cc:844] Serialized 29 byte log entry
0114 02:54:24.318518 (+ 274us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:24.318606 (+ 88us) write_op.cc:454] Released schema lock
0114 02:54:24.318647 (+ 41us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":2895744,"child_traces":[["op",{"apply.queue_time_us":7333,"num_ops":2,"prepare.queue_time_us":2414,"prepare.run_cpu_time_us":1696,"prepare.run_wall_time_us":1696,"raft.queue_time_us":1872,"raft.run_cpu_time_us":14,"raft.run_wall_time_us":14,"replication_time_us":127695,"spinlock_wait_cycles":18176}]]}
I20250114 02:54:24.380620 1207 ts_tablet_manager.cc:1428] T 42cee9e8e26147b593ed73fca6ac1183 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.094s user 0.007s sys 0.001s
I20250114 02:54:24.388595 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet 84738a6c78b24c1fb64013048763b18f (table default.Table_7058838b6bcb427786778096cf6f52b3 [id=b5d3461f83da46faaa772376a318b6b6]) successfully deleted
I20250114 02:54:24.389043 1207 tablet_bootstrap.cc:492] T ce7d6088fc884a248918d2348adcaccf P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:24.388994 1081 tablet_service.cc:1467] Processing CreateTablet for tablet 797d6cdf36704c98aa106aac87a945ca (DEFAULT_TABLE table=default.Table_5a9480c6474848d6b400e7dd274d15b0 [id=a17ad30ef32f4cbcb22b5a7d1c7096ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:24.389964 1081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 797d6cdf36704c98aa106aac87a945ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:24.393468 1084 tablet_service.cc:1467] Processing CreateTablet for tablet e7ca0bd1dbd542de9ee11e6c72522ae0 (DEFAULT_TABLE table=default.Table_7176743e4429467481ed67254643b9f9 [id=22e634dc594d4962b404424977f2cd74]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:24.394398 1084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7ca0bd1dbd542de9ee11e6c72522ae0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:24.398805 1086 tablet_service.cc:1467] Processing CreateTablet for tablet 97de018391f84b0ebd3130861681b7b5 (DEFAULT_TABLE table=default.Table_78eb79361f2c42538d83e9edcf6b51dc [id=2986220e36f94711ac450982e332f0f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:24.398800 1083 tablet_service.cc:1467] Processing CreateTablet for tablet 45301f8823ed4ec494e792d41f14f71f (DEFAULT_TABLE table=default.Table_8c98f977242949aa90be45b2f66d5679 [id=f203a9af37fa428aabc51c3244a86eac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:24.399893 1086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97de018391f84b0ebd3130861681b7b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:24.402560 1083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45301f8823ed4ec494e792d41f14f71f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:24.406963 793 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f1ea1608213492c9a216757896ff2a4 with delete_type TABLET_DATA_DELETED (replaced by tablet 550e84ac4d1b4ef58dfdea6e552ee940) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:24.408485 1594 tablet_replica.cc:331] T 0f1ea1608213492c9a216757896ff2a4 P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:24.426379 1207 tablet_bootstrap.cc:654] T ce7d6088fc884a248918d2348adcaccf P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
W20250114 02:54:23.989562 898 leader_election.cc:343] T ddfbab181b1f474192c664cd0316037f P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
W20250114 02:54:24.444828 898 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.67:45749, user_credentials={real_user=slave}} blocked reactor thread for 455110us
I20250114 02:54:24.447055 1206 tablet_bootstrap.cc:654] T ce7d6088fc884a248918d2348adcaccf P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:24.447402 814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42cee9e8e26147b593ed73fca6ac1183" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:24.448062 814 raft_consensus.cc:2463] T 42cee9e8e26147b593ed73fca6ac1183 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:24.448325 1102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42cee9e8e26147b593ed73fca6ac1183" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:24.448952 1102 raft_consensus.cc:2463] T 42cee9e8e26147b593ed73fca6ac1183 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:24.449061 813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46360f726fad4534a2daf67581346fc1" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:24.449808 1594 raft_consensus.cc:2238] T 0f1ea1608213492c9a216757896ff2a4 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:24.449913 813 raft_consensus.cc:2388] T 46360f726fad4534a2daf67581346fc1 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 525198593c8e47afaee54e74267b87e3 in current term 1: Already voted for candidate 90d6c2c58bb54e0eafb365625ac33871 in this term.
I20250114 02:54:24.450277 1594 raft_consensus.cc:2267] T 0f1ea1608213492c9a216757896ff2a4 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:24.453104 1099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46360f726fad4534a2daf67581346fc1" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:24.458983 1741 consensus_queue.cc:1035] T d3fb28717c6f4844af84bb817fe7dfc4 P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:54:24.466003 1594 ts_tablet_manager.cc:1905] T 0f1ea1608213492c9a216757896ff2a4 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:24.471810 1595 ts_tablet_manager.cc:1918] T 0f1ea1608213492c9a216757896ff2a4 P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:24.474126 1206 tablet_bootstrap.cc:492] T ce7d6088fc884a248918d2348adcaccf P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:24.479547 1948 consensus_queue.cc:1035] T f2b6812e8c1d44529d0284be490df648 P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:24.480425 1206 ts_tablet_manager.cc:1397] T ce7d6088fc884a248918d2348adcaccf P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.112s user 0.004s sys 0.007s
I20250114 02:54:24.485116 1595 log.cc:1198] T 0f1ea1608213492c9a216757896ff2a4 P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/0f1ea1608213492c9a216757896ff2a4
I20250114 02:54:24.489816 1595 ts_tablet_manager.cc:1939] T 0f1ea1608213492c9a216757896ff2a4 P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
I20250114 02:54:24.500860 462 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_a7de17bb754b4696bb8a0b05175061ad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:24.502339 462 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a7de17bb754b4696bb8a0b05175061ad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:24.512888 1206 raft_consensus.cc:357] T ce7d6088fc884a248918d2348adcaccf P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:24.516726 1206 raft_consensus.cc:383] T ce7d6088fc884a248918d2348adcaccf P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:24.517549 1206 raft_consensus.cc:738] T ce7d6088fc884a248918d2348adcaccf P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:24.518982 1206 consensus_queue.cc:260] T ce7d6088fc884a248918d2348adcaccf P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:24.529218 462 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_c62ee80bacab484cb74431dcb4723918"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:24.530771 462 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c62ee80bacab484cb74431dcb4723918 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:24.532546 1206 ts_tablet_manager.cc:1428] T ce7d6088fc884a248918d2348adcaccf P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.052s user 0.007s sys 0.001s
I20250114 02:54:24.534482 1206 tablet_bootstrap.cc:492] T e15e6e52d7294aeeb4463b3a76022ec3 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:24.375433 469 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 6448eea21ab2422c97da2494658ded3f (table default.Table_0931c459ae794d2091fcc0f51d031548 [id=e279c31bd63b40b9ade03a3b928bf153]). Expected version 2 got 0
I20250114 02:54:24.537293 469 catalog_manager.cc:5526] T f2b6812e8c1d44529d0284be490df648 P 819f9e3412e54d44abb11d8b3ca877c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65). New cstate: current_term: 1 leader_uuid: "819f9e3412e54d44abb11d8b3ca877c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:24.538758 469 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 044832851f804208b0a371eb0857dbf3 (table default.Table_0518d8bed7304c86aff32e05e6f07c0d [id=515a2fe3f1e84e82bccc02ef5cecef1b]). Expected version 1 got 0
I20250114 02:54:24.556354 1206 tablet_bootstrap.cc:654] T e15e6e52d7294aeeb4463b3a76022ec3 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:24.564944 1594 ts_tablet_manager.cc:1918] T 0f1ea1608213492c9a216757896ff2a4 P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:24.568550 1594 log.cc:1198] T 0f1ea1608213492c9a216757896ff2a4 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/0f1ea1608213492c9a216757896ff2a4
I20250114 02:54:24.569696 1594 ts_tablet_manager.cc:1939] T 0f1ea1608213492c9a216757896ff2a4 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
W20250114 02:54:24.576287 750 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.0.107.125:46837, user_credentials={real_user=slave}} blocked reactor thread for 68240.8us
I20250114 02:54:24.590282 1206 tablet_bootstrap.cc:492] T e15e6e52d7294aeeb4463b3a76022ec3 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:24.592537 1206 ts_tablet_manager.cc:1397] T e15e6e52d7294aeeb4463b3a76022ec3 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.058s user 0.010s sys 0.004s
I20250114 02:54:24.595157 1206 raft_consensus.cc:357] T e15e6e52d7294aeeb4463b3a76022ec3 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:24.599416 1206 raft_consensus.cc:383] T e15e6e52d7294aeeb4463b3a76022ec3 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:24.599800 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet 0f1ea1608213492c9a216757896ff2a4 (table default.Table_ae6b6fdfc1834cfb8089c860aedcc964 [id=7da003630a7a4e269d54fb2af8a3d4b5]) successfully deleted
I20250114 02:54:24.600196 1206 raft_consensus.cc:738] T e15e6e52d7294aeeb4463b3a76022ec3 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:24.600956 1085 tablet_service.cc:1467] Processing CreateTablet for tablet 038ea1ea231e490ba27ecd5430883058 (DEFAULT_TABLE table=default.Table_b41debc5d26d4d5aa95793afabcec087 [id=a6825ddf945843ffaf897d6ca6b62923]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:24.600947 1206 consensus_queue.cc:260] T e15e6e52d7294aeeb4463b3a76022ec3 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:24.601945 1085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 038ea1ea231e490ba27ecd5430883058. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:24.610227 789 tablet_service.cc:1467] Processing CreateTablet for tablet e7ca0bd1dbd542de9ee11e6c72522ae0 (DEFAULT_TABLE table=default.Table_7176743e4429467481ed67254643b9f9 [id=22e634dc594d4962b404424977f2cd74]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:24.611351 789 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7ca0bd1dbd542de9ee11e6c72522ae0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:24.611896 796 tablet_service.cc:1467] Processing CreateTablet for tablet 038ea1ea231e490ba27ecd5430883058 (DEFAULT_TABLE table=default.Table_b41debc5d26d4d5aa95793afabcec087 [id=a6825ddf945843ffaf897d6ca6b62923]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:24.612828 796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 038ea1ea231e490ba27ecd5430883058. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:24.613056 788 tablet_service.cc:1467] Processing CreateTablet for tablet 45301f8823ed4ec494e792d41f14f71f (DEFAULT_TABLE table=default.Table_8c98f977242949aa90be45b2f66d5679 [id=f203a9af37fa428aabc51c3244a86eac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:24.614352 788 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45301f8823ed4ec494e792d41f14f71f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:24.616881 793 tablet_service.cc:1467] Processing CreateTablet for tablet 97de018391f84b0ebd3130861681b7b5 (DEFAULT_TABLE table=default.Table_78eb79361f2c42538d83e9edcf6b51dc [id=2986220e36f94711ac450982e332f0f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:24.539451 469 catalog_manager.cc:5526] T d3fb28717c6f4844af84bb817fe7dfc4 P 819f9e3412e54d44abb11d8b3ca877c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65). New cstate: current_term: 1 leader_uuid: "819f9e3412e54d44abb11d8b3ca877c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:24.617781 793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97de018391f84b0ebd3130861681b7b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:24.635617 1820 consensus_queue.cc:1035] T 0304df55431e42e1a999ed17dc35e6c0 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20250114 02:54:24.654551 1206 ts_tablet_manager.cc:1428] T e15e6e52d7294aeeb4463b3a76022ec3 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.061s user 0.001s sys 0.004s
I20250114 02:54:24.655431 1206 tablet_bootstrap.cc:492] T 7774c2087422492696a58e0075921ae7 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:24.658012 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet 0f1ea1608213492c9a216757896ff2a4 (table default.Table_ae6b6fdfc1834cfb8089c860aedcc964 [id=7da003630a7a4e269d54fb2af8a3d4b5]) successfully deleted
I20250114 02:54:24.277487 1205 consensus_queue.cc:260] T 5ac7708550954a209e2a8d217c3b39bd P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:24.676506 1958 tablet.cc:1721] T 6448eea21ab2422c97da2494658ded3f P 819f9e3412e54d44abb11d8b3ca877c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250114 02:54:24.682788 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet 66bfa92db9874750b3c340e31a06c412 (table default.Table_b0cd93e5a498450e84fb0f1ba4be4505 [id=c70751b59bfc4c6fa60d0e8bb40d5f36]) successfully deleted
W20250114 02:54:24.683513 456 catalog_manager.cc:4852] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet e15e6e52d7294aeeb4463b3a76022ec3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e15e6e52d7294aeeb4463b3a76022ec3 already in progress: creating tablet
I20250114 02:54:24.684870 939 tablet_service.cc:1467] Processing CreateTablet for tablet e06daadd346d45969ad5efcf03bc0870 (DEFAULT_TABLE table=default.Table_1a9735a294c2436e84ec9db5e0aa44e4 [id=6922b5f98cc14e1d8a7e745211aa7b45]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:24.686412 939 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e06daadd346d45969ad5efcf03bc0870. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:24.687669 1206 tablet_bootstrap.cc:654] T 7774c2087422492696a58e0075921ae7 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:24.687038 465 catalog_manager.cc:5526] T 71bd00f7d8ac41839f36fd73997631ad P 525198593c8e47afaee54e74267b87e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 525198593c8e47afaee54e74267b87e3 (127.0.107.66). New cstate: current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:24.688789 465 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 55b68586ef8b45439a7dd8dcd2efcd79 (table default.Table_66f2263162f64250a0b49073abe4bd4c [id=c73ede1a5f664d8eb204fb9a012f8f7a]). Expected version 1 got 0
I20250114 02:54:24.689260 934 tablet_service.cc:1467] Processing CreateTablet for tablet ed109b49f20642f988161a3df7872419 (DEFAULT_TABLE table=default.Table_529c38ed585343acb0f42ac60f171b59 [id=af3fec2e3d0243bfbed7aa314a9732a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:24.690438 934 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed109b49f20642f988161a3df7872419. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:24.690824 465 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 0304df55431e42e1a999ed17dc35e6c0 (table default.Table_4c3843233dda407395ab9652d5870bc6 [id=0d173fdb502f482fb86b6d3e5fdc66f1]). Expected version 1 got 0
I20250114 02:54:24.691486 465 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 6448eea21ab2422c97da2494658ded3f (table default.Table_0931c459ae794d2091fcc0f51d031548 [id=e279c31bd63b40b9ade03a3b928bf153]). Expected version 2 got 0
I20250114 02:54:24.692010 941 tablet_service.cc:1467] Processing CreateTablet for tablet 2e17f619ed2745078541cc1c924aff96 (DEFAULT_TABLE table=default.Table_1142a1a45945449ea16f5501c3adeb8b [id=b124b43245d14d7daebf8d33882c9124]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:24.692685 942 tablet_service.cc:1514] Processing DeleteTablet for tablet 84738a6c78b24c1fb64013048763b18f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:23 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:24.693248 941 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e17f619ed2745078541cc1c924aff96. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:24.704401 1948 raft_consensus.cc:491] T ce7d6088fc884a248918d2348adcaccf P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:24.704905 1948 raft_consensus.cc:513] T ce7d6088fc884a248918d2348adcaccf P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:24.708565 1099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce7d6088fc884a248918d2348adcaccf" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
W20250114 02:54:24.709930 753 leader_election.cc:343] T ce7d6088fc884a248918d2348adcaccf P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:24.711781 933 tablet_service.cc:1467] Processing CreateTablet for tablet 797d6cdf36704c98aa106aac87a945ca (DEFAULT_TABLE table=default.Table_5a9480c6474848d6b400e7dd274d15b0 [id=a17ad30ef32f4cbcb22b5a7d1c7096ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:24.712983 933 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 797d6cdf36704c98aa106aac87a945ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:24.716497 1957 tablet.cc:1721] T 6448eea21ab2422c97da2494658ded3f P 90d6c2c58bb54e0eafb365625ac33871: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250114 02:54:24.721133 936 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f1ea1608213492c9a216757896ff2a4 with delete_type TABLET_DATA_DELETED (replaced by tablet 550e84ac4d1b4ef58dfdea6e552ee940) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:24.722545 937 tablet_service.cc:1467] Processing CreateTablet for tablet e7ca0bd1dbd542de9ee11e6c72522ae0 (DEFAULT_TABLE table=default.Table_7176743e4429467481ed67254643b9f9 [id=22e634dc594d4962b404424977f2cd74]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:24.723611 937 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7ca0bd1dbd542de9ee11e6c72522ae0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:24.725045 940 tablet_service.cc:1467] Processing CreateTablet for tablet 45301f8823ed4ec494e792d41f14f71f (DEFAULT_TABLE table=default.Table_8c98f977242949aa90be45b2f66d5679 [id=f203a9af37fa428aabc51c3244a86eac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:24.726084 940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45301f8823ed4ec494e792d41f14f71f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:24.729065 938 tablet_service.cc:1467] Processing CreateTablet for tablet 038ea1ea231e490ba27ecd5430883058 (DEFAULT_TABLE table=default.Table_b41debc5d26d4d5aa95793afabcec087 [id=a6825ddf945843ffaf897d6ca6b62923]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:24.730053 938 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 038ea1ea231e490ba27ecd5430883058. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:24.733376 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46360f726fad4534a2daf67581346fc1" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:24.734280 962 raft_consensus.cc:2463] T 46360f726fad4534a2daf67581346fc1 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 0.
I20250114 02:54:24.736146 959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46360f726fad4534a2daf67581346fc1" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
I20250114 02:54:24.736744 959 raft_consensus.cc:3054] T 46360f726fad4534a2daf67581346fc1 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:24.736800 1948 leader_election.cc:290] T ce7d6088fc884a248918d2348adcaccf P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:24.741536 1948 raft_consensus.cc:491] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:24.742005 1948 raft_consensus.cc:513] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:24.745375 465 catalog_manager.cc:5526] T 044832851f804208b0a371eb0857dbf3 P 525198593c8e47afaee54e74267b87e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 525198593c8e47afaee54e74267b87e3 (127.0.107.66). New cstate: current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:24.746574 465 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 044832851f804208b0a371eb0857dbf3 (table default.Table_0518d8bed7304c86aff32e05e6f07c0d [id=515a2fe3f1e84e82bccc02ef5cecef1b]). Expected version 1 got 0
I20250114 02:54:24.747306 1948 leader_election.cc:290] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:24.749574 1206 tablet_bootstrap.cc:492] T 7774c2087422492696a58e0075921ae7 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:24.749955 1206 ts_tablet_manager.cc:1397] T 7774c2087422492696a58e0075921ae7 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.095s user 0.009s sys 0.000s
I20250114 02:54:24.755225 1820 consensus_queue.cc:1035] T 0304df55431e42e1a999ed17dc35e6c0 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:24.756637 1206 raft_consensus.cc:357] T 7774c2087422492696a58e0075921ae7 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:24.758294 1206 raft_consensus.cc:383] T 7774c2087422492696a58e0075921ae7 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:24.758564 1206 raft_consensus.cc:738] T 7774c2087422492696a58e0075921ae7 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
W20250114 02:54:24.758772 1040 consensus_peers.cc:487] T bc46815b4803418c95ea9c039a4fae88 P 90d6c2c58bb54e0eafb365625ac33871 -> Peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Couldn't send request to peer 525198593c8e47afaee54e74267b87e3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bc46815b4803418c95ea9c039a4fae88. This is attempt 1: this message will repeat every 5th retry.
I20250114 02:54:24.760262 1206 consensus_queue.cc:260] T 7774c2087422492696a58e0075921ae7 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:24.764660 1102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0bd9bd5a17d418d8082ed0ebd1bf4cc" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:24.765295 1102 raft_consensus.cc:2463] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 0.
I20250114 02:54:24.766932 936 tablet_service.cc:1467] Processing CreateTablet for tablet 97de018391f84b0ebd3130861681b7b5 (DEFAULT_TABLE table=default.Table_78eb79361f2c42538d83e9edcf6b51dc [id=2986220e36f94711ac450982e332f0f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:24.762569 1206 ts_tablet_manager.cc:1428] T 7774c2087422492696a58e0075921ae7 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20250114 02:54:24.769259 753 leader_election.cc:304] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:24.778100 1948 raft_consensus.cc:2798] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:24.778525 1948 raft_consensus.cc:491] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:24.778896 1948 raft_consensus.cc:3054] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:24.786770 1741 raft_consensus.cc:491] T 7774c2087422492696a58e0075921ae7 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:24.787252 1741 raft_consensus.cc:513] T 7774c2087422492696a58e0075921ae7 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:24.791218 1099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7774c2087422492696a58e0075921ae7" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
W20250114 02:54:24.793218 753 leader_election.cc:343] T 7774c2087422492696a58e0075921ae7 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:24.793730 1741 leader_election.cc:290] T 7774c2087422492696a58e0075921ae7 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:24.794618 1948 raft_consensus.cc:513] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:24.798434 1206 tablet_bootstrap.cc:492] T 92cd8eccacf34ab7a66b3bbc69df3697 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:24.807178 1205 ts_tablet_manager.cc:1428] T 5ac7708550954a209e2a8d217c3b39bd P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.534s user 0.008s sys 0.001s
W20250114 02:54:24.807675 1205 ts_tablet_manager.cc:1460] T 5ac7708550954a209e2a8d217c3b39bd P 525198593c8e47afaee54e74267b87e3: Tablet startup took 534ms
I20250114 02:54:24.809731 1207 tablet_bootstrap.cc:492] T ce7d6088fc884a248918d2348adcaccf P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:24.810305 1207 ts_tablet_manager.cc:1397] T ce7d6088fc884a248918d2348adcaccf P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.421s user 0.013s sys 0.007s
I20250114 02:54:24.810482 936 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97de018391f84b0ebd3130861681b7b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:24.812619 1207 raft_consensus.cc:357] T ce7d6088fc884a248918d2348adcaccf P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:24.813290 1207 raft_consensus.cc:383] T ce7d6088fc884a248918d2348adcaccf P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:24.813522 1207 raft_consensus.cc:738] T ce7d6088fc884a248918d2348adcaccf P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:24.814083 1207 consensus_queue.cc:260] T ce7d6088fc884a248918d2348adcaccf P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
W20250114 02:54:24.808503 1205 ts_tablet_manager.cc:1462] T 5ac7708550954a209e2a8d217c3b39bd P 525198593c8e47afaee54e74267b87e3: Trace:
0114 02:54:18.535901 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:18.536022 (+ 121us) service_pool.cc:225] Handling call
0114 02:54:18.604142 (+ 68120us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0114 02:54:18.604250 (+ 108us) ts_tablet_manager.cc:676] Creating new metadata...
0114 02:54:18.628186 (+ 23936us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:18.704953 (+ 76767us) tablet_replica.cc:206] Creating consensus instance
0114 02:54:18.705269 (+ 316us) inbound_call.cc:173] Queueing success response
0114 02:54:24.080132 (+5374863us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0114 02:54:24.273517 (+193385us) ts_tablet_manager.cc:1429] Starting tablet replica
0114 02:54:24.274372 (+ 855us) tablet_replica.cc:260] Starting instrumentation
0114 02:54:24.274781 (+ 409us) tablet_replica.cc:279] Starting consensus
Metrics: {"mutex_wait_us":98198,"raft.queue_time_us":96886,"raft.run_cpu_time_us":227,"raft.run_wall_time_us":228,"tablet-open.queue_time_us":5374918}
I20250114 02:54:24.823338 1205 tablet_bootstrap.cc:492] T 84738a6c78b24c1fb64013048763b18f P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:24.851312 1099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0bd9bd5a17d418d8082ed0ebd1bf4cc" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:24.852093 1099 raft_consensus.cc:3054] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:24.861305 1099 raft_consensus.cc:2463] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 1.
I20250114 02:54:24.862751 753 leader_election.cc:304] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:24.863910 1961 raft_consensus.cc:2798] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:24.864497 1961 raft_consensus.cc:695] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 LEADER]: Becoming Leader. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Running, Role: LEADER
I20250114 02:54:24.865435 1961 consensus_queue.cc:237] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:24.875044 1948 leader_election.cc:290] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Requested vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:24.881705 1965 tablet_replica.cc:331] T 0f1ea1608213492c9a216757896ff2a4 P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:24.883219 1965 raft_consensus.cc:2238] T 0f1ea1608213492c9a216757896ff2a4 P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Raft consensus shutting down.
I20250114 02:54:24.884723 1965 raft_consensus.cc:2267] T 0f1ea1608213492c9a216757896ff2a4 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:24.887814 1965 ts_tablet_manager.cc:1905] T 0f1ea1608213492c9a216757896ff2a4 P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:24.894368 1207 ts_tablet_manager.cc:1428] T ce7d6088fc884a248918d2348adcaccf P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.084s user 0.005s sys 0.000s
I20250114 02:54:24.836023 959 raft_consensus.cc:2463] T 46360f726fad4534a2daf67581346fc1 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1.
W20250114 02:54:24.905768 456 catalog_manager.cc:4852] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet 84738a6c78b24c1fb64013048763b18f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 84738a6c78b24c1fb64013048763b18f already in progress: creating tablet
I20250114 02:54:24.903992 1206 tablet_bootstrap.cc:654] T 92cd8eccacf34ab7a66b3bbc69df3697 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:24.929261 1206 tablet_bootstrap.cc:492] T 92cd8eccacf34ab7a66b3bbc69df3697 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:24.931639 1206 ts_tablet_manager.cc:1397] T 92cd8eccacf34ab7a66b3bbc69df3697 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.133s user 0.009s sys 0.002s
I20250114 02:54:24.937872 1207 tablet_bootstrap.cc:492] T ddfbab181b1f474192c664cd0316037f P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:24.937674 1944 raft_consensus.cc:491] T ce7d6088fc884a248918d2348adcaccf P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:24.938287 1944 raft_consensus.cc:513] T ce7d6088fc884a248918d2348adcaccf P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:24.940466 1206 raft_consensus.cc:357] T 92cd8eccacf34ab7a66b3bbc69df3697 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:24.941108 1206 raft_consensus.cc:383] T 92cd8eccacf34ab7a66b3bbc69df3697 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:24.941354 1206 raft_consensus.cc:738] T 92cd8eccacf34ab7a66b3bbc69df3697 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:24.952006 1207 tablet_bootstrap.cc:654] T ddfbab181b1f474192c664cd0316037f P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:24.954183 1206 consensus_queue.cc:260] T 92cd8eccacf34ab7a66b3bbc69df3697 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:24.956307 1206 ts_tablet_manager.cc:1428] T 92cd8eccacf34ab7a66b3bbc69df3697 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20250114 02:54:24.970194 1206 tablet_bootstrap.cc:492] T ddfbab181b1f474192c664cd0316037f P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:24.964275 1966 consensus_queue.cc:1035] T 46360f726fad4534a2daf67581346fc1 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:24.994781 1206 tablet_bootstrap.cc:654] T ddfbab181b1f474192c664cd0316037f P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:24.997565 1961 raft_consensus.cc:491] T 5ac7708550954a209e2a8d217c3b39bd P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:24.998116 1961 raft_consensus.cc:513] T 5ac7708550954a209e2a8d217c3b39bd P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:25.001034 1961 leader_election.cc:290] T 5ac7708550954a209e2a8d217c3b39bd P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:25.002969 1099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ac7708550954a209e2a8d217c3b39bd" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
W20250114 02:54:25.004748 753 leader_election.cc:343] T 5ac7708550954a209e2a8d217c3b39bd P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:25.015928 1966 raft_consensus.cc:491] T ce7d6088fc884a248918d2348adcaccf P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:25.016762 1966 raft_consensus.cc:513] T ce7d6088fc884a248918d2348adcaccf P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:25.018710 1966 leader_election.cc:290] T ce7d6088fc884a248918d2348adcaccf P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:25.019959 1207 tablet_bootstrap.cc:492] T ddfbab181b1f474192c664cd0316037f P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:25.020395 817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce7d6088fc884a248918d2348adcaccf" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:25.021149 817 raft_consensus.cc:2463] T ce7d6088fc884a248918d2348adcaccf P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 0.
I20250114 02:54:25.022297 1042 leader_election.cc:304] T ce7d6088fc884a248918d2348adcaccf P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:25.023500 1966 raft_consensus.cc:2798] T ce7d6088fc884a248918d2348adcaccf P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:25.023892 1966 raft_consensus.cc:491] T ce7d6088fc884a248918d2348adcaccf P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:25.024296 1966 raft_consensus.cc:3054] T ce7d6088fc884a248918d2348adcaccf P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:25.033046 1207 ts_tablet_manager.cc:1397] T ddfbab181b1f474192c664cd0316037f P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.095s user 0.014s sys 0.000s
I20250114 02:54:25.035490 1207 raft_consensus.cc:357] T ddfbab181b1f474192c664cd0316037f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:25.037339 1944 leader_election.cc:290] T ce7d6088fc884a248918d2348adcaccf P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:25.038928 1205 tablet_bootstrap.cc:654] T 84738a6c78b24c1fb64013048763b18f P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:25.039301 1206 tablet_bootstrap.cc:492] T ddfbab181b1f474192c664cd0316037f P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:25.039741 1206 ts_tablet_manager.cc:1397] T ddfbab181b1f474192c664cd0316037f P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.070s user 0.009s sys 0.000s
I20250114 02:54:25.042338 1206 raft_consensus.cc:357] T ddfbab181b1f474192c664cd0316037f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:25.043063 1206 raft_consensus.cc:383] T ddfbab181b1f474192c664cd0316037f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:25.043365 1206 raft_consensus.cc:738] T ddfbab181b1f474192c664cd0316037f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:25.049297 1206 consensus_queue.cc:260] T ddfbab181b1f474192c664cd0316037f P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:25.058712 960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce7d6088fc884a248918d2348adcaccf" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:25.059321 960 raft_consensus.cc:2463] T ce7d6088fc884a248918d2348adcaccf P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 0.
I20250114 02:54:25.061975 959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce7d6088fc884a248918d2348adcaccf" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:25.063154 959 raft_consensus.cc:2463] T ce7d6088fc884a248918d2348adcaccf P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 0.
I20250114 02:54:25.074973 959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0bd9bd5a17d418d8082ed0ebd1bf4cc" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:25.076556 959 raft_consensus.cc:2463] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 0.
I20250114 02:54:25.077991 1206 ts_tablet_manager.cc:1428] T ddfbab181b1f474192c664cd0316037f P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.038s user 0.003s sys 0.005s
W20250114 02:54:25.080358 455 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.124:39105, user_credentials={real_user=slave}} blocked reactor thread for 180373us
I20250114 02:54:25.082430 1961 consensus_queue.cc:1035] T ca986f81ed6848a89588366f3f400eec P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:25.087522 959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7774c2087422492696a58e0075921ae7" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:25.094996 1206 tablet_bootstrap.cc:492] T edc69b66bea54625a4b15867b75986b7 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
W20250114 02:54:25.099090 455 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 6e858a8449444ef999b9bbef612249d3 (table default.Table_5a9480c6474848d6b400e7dd274d15b0 [id=a17ad30ef32f4cbcb22b5a7d1c7096ff], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 6e858a8449444ef999b9bbef612249d3 consensus state has no leader
W20250114 02:54:25.099500 455 consensus_peers.cc:487] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750 -> Peer 87b3db09ac87436e9a60d6de5fedaff5 (127.0.107.124:39105): Couldn't send request to peer 87b3db09ac87436e9a60d6de5fedaff5. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.124:39105: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20250114 02:54:25.114281 938 tablet_service.cc:1467] Processing CreateTablet for tablet ab9a85db127b4d82ae1161717ed6006a (DEFAULT_TABLE table=default.Table_651d4cc08fe74c4aa0bbcf876d11b8e8 [id=e9b187184b894e3f916e268816339767]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:25.114713 1206 tablet_bootstrap.cc:654] T edc69b66bea54625a4b15867b75986b7 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:25.119004 796 tablet_service.cc:1467] Processing CreateTablet for tablet ab9a85db127b4d82ae1161717ed6006a (DEFAULT_TABLE table=default.Table_651d4cc08fe74c4aa0bbcf876d11b8e8 [id=e9b187184b894e3f916e268816339767]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:25.119763 942 tablet_service.cc:1467] Processing CreateTablet for tablet 2d4eaf90be6d497d8a3c4d9ac418f4e9 (DEFAULT_TABLE table=default.Table_73ec0be3adc440d788c369d80d7bbefa [id=e6bda3701397455e90500d3a956df88b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:25.120261 796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab9a85db127b4d82ae1161717ed6006a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:25.120780 942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d4eaf90be6d497d8a3c4d9ac418f4e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:25.126113 789 tablet_service.cc:1467] Processing CreateTablet for tablet 2d4eaf90be6d497d8a3c4d9ac418f4e9 (DEFAULT_TABLE table=default.Table_73ec0be3adc440d788c369d80d7bbefa [id=e6bda3701397455e90500d3a956df88b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:25.127138 789 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d4eaf90be6d497d8a3c4d9ac418f4e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:25.129905 788 tablet_service.cc:1467] Processing CreateTablet for tablet a2c3f3a89b3d47f1b0dfacfac8218805 (DEFAULT_TABLE table=default.Table_ebe5f78fc2dc4adeacda49c55c5b2147 [id=62526be71bc54ef796c1cabfa7ee0da3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:25.130918 788 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2c3f3a89b3d47f1b0dfacfac8218805. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:25.130977 941 tablet_service.cc:1467] Processing CreateTablet for tablet a2c3f3a89b3d47f1b0dfacfac8218805 (DEFAULT_TABLE table=default.Table_ebe5f78fc2dc4adeacda49c55c5b2147 [id=62526be71bc54ef796c1cabfa7ee0da3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:25.131875 941 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2c3f3a89b3d47f1b0dfacfac8218805. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:25.133342 793 tablet_service.cc:1467] Processing CreateTablet for tablet 6d741bfd00784d2d830be3ebe50dc7c4 (DEFAULT_TABLE table=default.Table_d1de82caa5b94791b6c7dfed1c376e8a [id=03c775a54e9e443cbc4fab853bd6d49c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:25.134217 793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d741bfd00784d2d830be3ebe50dc7c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:25.134622 933 tablet_service.cc:1467] Processing CreateTablet for tablet 6d741bfd00784d2d830be3ebe50dc7c4 (DEFAULT_TABLE table=default.Table_d1de82caa5b94791b6c7dfed1c376e8a [id=03c775a54e9e443cbc4fab853bd6d49c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:25.135587 933 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d741bfd00784d2d830be3ebe50dc7c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:25.139423 464 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet 6448eea21ab2422c97da2494658ded3f (table default.Table_0931c459ae794d2091fcc0f51d031548 [id=e279c31bd63b40b9ade03a3b928bf153]). Expected version 2 got 0
I20250114 02:54:25.142517 935 tablet_service.cc:1467] Processing CreateTablet for tablet 46eb13912f754bf1b1fe9d0ad429c3c1 (DEFAULT_TABLE table=default.Table_972e696cae0e4f05a434e5375f6767c5 [id=456f726bff3d46e3b454f9b10e84c3c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:25.143296 790 tablet_service.cc:1467] Processing CreateTablet for tablet 46eb13912f754bf1b1fe9d0ad429c3c1 (DEFAULT_TABLE table=default.Table_972e696cae0e4f05a434e5375f6767c5 [id=456f726bff3d46e3b454f9b10e84c3c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:25.143666 935 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46eb13912f754bf1b1fe9d0ad429c3c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:25.144421 790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46eb13912f754bf1b1fe9d0ad429c3c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:25.150279 797 tablet_service.cc:1467] Processing CreateTablet for tablet 550e84ac4d1b4ef58dfdea6e552ee940 (DEFAULT_TABLE table=default.Table_ae6b6fdfc1834cfb8089c860aedcc964 [id=7da003630a7a4e269d54fb2af8a3d4b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:25.151198 797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 550e84ac4d1b4ef58dfdea6e552ee940. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:25.166774 1206 tablet_bootstrap.cc:492] T edc69b66bea54625a4b15867b75986b7 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:25.167182 1206 ts_tablet_manager.cc:1397] T edc69b66bea54625a4b15867b75986b7 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.072s user 0.005s sys 0.006s
I20250114 02:54:25.167063 932 tablet_service.cc:1467] Processing CreateTablet for tablet 550e84ac4d1b4ef58dfdea6e552ee940 (DEFAULT_TABLE table=default.Table_ae6b6fdfc1834cfb8089c860aedcc964 [id=7da003630a7a4e269d54fb2af8a3d4b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:25.168210 932 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 550e84ac4d1b4ef58dfdea6e552ee940. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:25.147505 464 catalog_manager.cc:5526] T 0304df55431e42e1a999ed17dc35e6c0 P 90d6c2c58bb54e0eafb365625ac33871 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67). New cstate: current_term: 1 leader_uuid: "90d6c2c58bb54e0eafb365625ac33871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } }
I20250114 02:54:25.171295 464 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet 0304df55431e42e1a999ed17dc35e6c0 (table default.Table_4c3843233dda407395ab9652d5870bc6 [id=0d173fdb502f482fb86b6d3e5fdc66f1]). Expected version 1 got 0
I20250114 02:54:25.172864 464 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet 55b68586ef8b45439a7dd8dcd2efcd79 (table default.Table_66f2263162f64250a0b49073abe4bd4c [id=c73ede1a5f664d8eb204fb9a012f8f7a]). Expected version 1 got 0
I20250114 02:54:25.173915 464 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet 044832851f804208b0a371eb0857dbf3 (table default.Table_0518d8bed7304c86aff32e05e6f07c0d [id=515a2fe3f1e84e82bccc02ef5cecef1b]). Expected version 1 got 0
I20250114 02:54:25.189253 1206 raft_consensus.cc:357] T edc69b66bea54625a4b15867b75986b7 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:25.189932 1206 raft_consensus.cc:383] T edc69b66bea54625a4b15867b75986b7 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:25.190181 1206 raft_consensus.cc:738] T edc69b66bea54625a4b15867b75986b7 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:25.190455 957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0bd9bd5a17d418d8082ed0ebd1bf4cc" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
I20250114 02:54:25.190739 1206 consensus_queue.cc:260] T edc69b66bea54625a4b15867b75986b7 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:25.191152 957 raft_consensus.cc:3054] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:25.116289 1841 consensus_queue.cc:1035] T 46360f726fad4534a2daf67581346fc1 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:25.119259 1085 tablet_service.cc:1467] Processing CreateTablet for tablet ab9a85db127b4d82ae1161717ed6006a (DEFAULT_TABLE table=default.Table_651d4cc08fe74c4aa0bbcf876d11b8e8 [id=e9b187184b894e3f916e268816339767]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:25.199044 1085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab9a85db127b4d82ae1161717ed6006a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:25.199213 1206 ts_tablet_manager.cc:1428] T edc69b66bea54625a4b15867b75986b7 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.031s user 0.007s sys 0.000s
I20250114 02:54:25.199307 938 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab9a85db127b4d82ae1161717ed6006a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:25.201247 1206 tablet_bootstrap.cc:492] T 38fe8829d98547fb907ef78099df3d7f P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:25.126880 1081 tablet_service.cc:1467] Processing CreateTablet for tablet 2d4eaf90be6d497d8a3c4d9ac418f4e9 (DEFAULT_TABLE table=default.Table_73ec0be3adc440d788c369d80d7bbefa [id=e6bda3701397455e90500d3a956df88b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:25.205335 1081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d4eaf90be6d497d8a3c4d9ac418f4e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:25.131107 1083 tablet_service.cc:1467] Processing CreateTablet for tablet a2c3f3a89b3d47f1b0dfacfac8218805 (DEFAULT_TABLE table=default.Table_ebe5f78fc2dc4adeacda49c55c5b2147 [id=62526be71bc54ef796c1cabfa7ee0da3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:25.209280 1083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2c3f3a89b3d47f1b0dfacfac8218805. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:25.135732 1084 tablet_service.cc:1467] Processing CreateTablet for tablet 6d741bfd00784d2d830be3ebe50dc7c4 (DEFAULT_TABLE table=default.Table_d1de82caa5b94791b6c7dfed1c376e8a [id=03c775a54e9e443cbc4fab853bd6d49c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:25.212960 1084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d741bfd00784d2d830be3ebe50dc7c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:25.146514 1086 tablet_service.cc:1467] Processing CreateTablet for tablet 46eb13912f754bf1b1fe9d0ad429c3c1 (DEFAULT_TABLE table=default.Table_972e696cae0e4f05a434e5375f6767c5 [id=456f726bff3d46e3b454f9b10e84c3c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:25.216864 1086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46eb13912f754bf1b1fe9d0ad429c3c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:25.197933 1207 raft_consensus.cc:383] T ddfbab181b1f474192c664cd0316037f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:25.217751 1207 raft_consensus.cc:738] T ddfbab181b1f474192c664cd0316037f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:25.179471 1082 tablet_service.cc:1467] Processing CreateTablet for tablet 550e84ac4d1b4ef58dfdea6e552ee940 (DEFAULT_TABLE table=default.Table_ae6b6fdfc1834cfb8089c860aedcc964 [id=7da003630a7a4e269d54fb2af8a3d4b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:25.221879 1082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 550e84ac4d1b4ef58dfdea6e552ee940. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:25.195349 1966 raft_consensus.cc:513] T ce7d6088fc884a248918d2348adcaccf P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:25.225528 814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce7d6088fc884a248918d2348adcaccf" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:25.226095 814 raft_consensus.cc:3054] T ce7d6088fc884a248918d2348adcaccf P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:25.233675 814 raft_consensus.cc:2463] T ce7d6088fc884a248918d2348adcaccf P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1.
I20250114 02:54:25.234830 1042 leader_election.cc:304] T ce7d6088fc884a248918d2348adcaccf P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:25.235633 1841 raft_consensus.cc:2798] T ce7d6088fc884a248918d2348adcaccf P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:25.236027 1841 raft_consensus.cc:695] T ce7d6088fc884a248918d2348adcaccf P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Becoming Leader. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Running, Role: LEADER
I20250114 02:54:25.236766 1841 consensus_queue.cc:237] T ce7d6088fc884a248918d2348adcaccf P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:25.225229 1207 consensus_queue.cc:260] T ddfbab181b1f474192c664cd0316037f P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:25.242400 1207 ts_tablet_manager.cc:1428] T ddfbab181b1f474192c664cd0316037f P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.209s user 0.006s sys 0.000s
I20250114 02:54:25.243532 1207 tablet_bootstrap.cc:492] T e15e6e52d7294aeeb4463b3a76022ec3 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:25.250674 462 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 5ac7708550954a209e2a8d217c3b39bd (table default.Table_ee814dd266e64286864db0eac9e11664 [id=13151866bf2b4f44b1dd5c92380a4d43]). Expected version 1 got 0
I20250114 02:54:25.237752 1966 leader_election.cc:290] T ce7d6088fc884a248918d2348adcaccf P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Requested vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
W20250114 02:54:25.220930 898 leader_election.cc:343] T ddfbab181b1f474192c664cd0316037f P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:25.274827 958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ac7708550954a209e2a8d217c3b39bd" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:25.275656 958 raft_consensus.cc:2463] T 5ac7708550954a209e2a8d217c3b39bd P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 0.
I20250114 02:54:25.282259 1957 tablet.cc:1721] T 55b68586ef8b45439a7dd8dcd2efcd79 P 90d6c2c58bb54e0eafb365625ac33871: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250114 02:54:25.282968 1741 raft_consensus.cc:491] T e15e6e52d7294aeeb4463b3a76022ec3 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:25.283385 1741 raft_consensus.cc:513] T e15e6e52d7294aeeb4463b3a76022ec3 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:25.284629 1958 tablet.cc:1721] T 55b68586ef8b45439a7dd8dcd2efcd79 P 819f9e3412e54d44abb11d8b3ca877c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250114 02:54:25.286562 1741 leader_election.cc:290] T e15e6e52d7294aeeb4463b3a76022ec3 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:25.287276 1099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e15e6e52d7294aeeb4463b3a76022ec3" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
W20250114 02:54:25.288561 753 leader_election.cc:343] T e15e6e52d7294aeeb4463b3a76022ec3 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:25.261761 898 leader_election.cc:304] T ddfbab181b1f474192c664cd0316037f P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 525198593c8e47afaee54e74267b87e3; no voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:25.299269 1944 raft_consensus.cc:2743] T ddfbab181b1f474192c664cd0316037f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:25.300014 1206 tablet_bootstrap.cc:654] T 38fe8829d98547fb907ef78099df3d7f P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
W20250114 02:54:25.304313 898 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.65:39397, user_credentials={real_user=slave}} blocked reactor thread for 83411.1us
I20250114 02:54:25.325891 898 leader_election.cc:304] T 42cee9e8e26147b593ed73fca6ac1183 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
I20250114 02:54:25.326848 1942 raft_consensus.cc:2798] T 42cee9e8e26147b593ed73fca6ac1183 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:25.327163 1942 raft_consensus.cc:491] T 42cee9e8e26147b593ed73fca6ac1183 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:25.327453 1942 raft_consensus.cc:3054] T 42cee9e8e26147b593ed73fca6ac1183 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:25.334096 1942 raft_consensus.cc:513] T 42cee9e8e26147b593ed73fca6ac1183 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:25.336298 1942 leader_election.cc:290] T 42cee9e8e26147b593ed73fca6ac1183 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Requested vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:25.354753 462 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet ce7d6088fc884a248918d2348adcaccf (table default.Table_428bd60f16824ac2b8bfaad6819f76e1 [id=35c5879da4b344118a377335fc6f83de]). Expected version 1 got 0
I20250114 02:54:25.355903 462 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 0304df55431e42e1a999ed17dc35e6c0 (table default.Table_4c3843233dda407395ab9652d5870bc6 [id=0d173fdb502f482fb86b6d3e5fdc66f1]). Expected version 1 got 0
I20250114 02:54:25.356900 462 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 6448eea21ab2422c97da2494658ded3f (table default.Table_0931c459ae794d2091fcc0f51d031548 [id=e279c31bd63b40b9ade03a3b928bf153]). Expected version 2 got 0
I20250114 02:54:25.357635 462 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 044832851f804208b0a371eb0857dbf3 (table default.Table_0518d8bed7304c86aff32e05e6f07c0d [id=515a2fe3f1e84e82bccc02ef5cecef1b]). Expected version 1 got 0
I20250114 02:54:25.359947 462 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 55b68586ef8b45439a7dd8dcd2efcd79 (table default.Table_66f2263162f64250a0b49073abe4bd4c [id=c73ede1a5f664d8eb204fb9a012f8f7a]). Expected version 1 got 0
I20250114 02:54:25.361002 960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce7d6088fc884a248918d2348adcaccf" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
I20250114 02:54:25.361747 960 raft_consensus.cc:3054] T ce7d6088fc884a248918d2348adcaccf P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:25.364982 467 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: default.Table_b624816662024c55a5f5ef5830b20b7c [id=d2ea84ea59ad4b03ad16bb3052a441f4] alter complete (version 2)
I20250114 02:54:25.366873 466 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet 321b9ad6d1ac4d1c80c7a01faf2015af
I20250114 02:54:25.369688 960 raft_consensus.cc:2463] T ce7d6088fc884a248918d2348adcaccf P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1.
I20250114 02:54:25.371162 466 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:33540 (request call id 234) took 2696 ms. Trace:
W20250114 02:54:25.371914 898 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.65:39397, user_credentials={real_user=slave}} blocked reactor thread for 52899.1us
I20250114 02:54:25.368306 471 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: default.Table_9676926a50eb42349a2d609b30566e9a [id=f38063ebf94d4084ab0174428503dc29] alter complete (version 2)
I20250114 02:54:25.371551 466 rpcz_store.cc:276] 0114 02:54:22.675090 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:23.842135 (+1167045us) service_pool.cc:225] Handling call
0114 02:54:23.843381 (+ 1246us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0114 02:54:23.885251 (+ 41870us) catalog_manager.cc:2753] Modifying in-memory table state
0114 02:54:23.885390 (+ 139us) catalog_manager.cc:2762] Locking tablets
0114 02:54:23.885901 (+ 511us) catalog_manager.cc:2776] Removing table and tablets from system table
0114 02:54:23.948760 (+ 62859us) catalog_manager.cc:2794] Removing table from by-name map
0114 02:54:23.970778 (+ 22018us) catalog_manager.cc:2810] Committing in-memory state
0114 02:54:25.366634 (+1395856us) catalog_manager.cc:2814] Aborting table tasks
0114 02:54:25.371060 (+ 4426us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:23.913447 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:23.913805 (+ 358us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:23.913827 (+ 22us) write_op.cc:435] Acquired schema lock
0114 02:54:23.913840 (+ 13us) tablet.cc:659] Decoding operations
0114 02:54:23.914129 (+ 289us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:23.914172 (+ 43us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:23.914194 (+ 22us) tablet.cc:682] Acquiring locks for 2 operations
0114 02:54:23.914429 (+ 235us) tablet.cc:698] Row locks acquired
0114 02:54:23.914443 (+ 14us) write_op.cc:260] PREPARE: finished
0114 02:54:23.914546 (+ 103us) write_op.cc:270] Start()
0114 02:54:23.914624 (+ 78us) write_op.cc:276] Timestamp: P: 1736823263914491 usec, L: 0
0114 02:54:23.914638 (+ 14us) op_driver.cc:348] REPLICATION: starting
0114 02:54:23.915076 (+ 438us) log.cc:844] Serialized 762 byte log entry
0114 02:54:23.943884 (+ 28808us) op_driver.cc:464] REPLICATION: finished
0114 02:54:23.946100 (+ 2216us) write_op.cc:301] APPLY: starting
0114 02:54:23.946146 (+ 46us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:23.946256 (+ 110us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:23.946357 (+ 101us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:23.946617 (+ 260us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:23.946797 (+ 180us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0114 02:54:23.946837 (+ 40us) write_op.cc:312] APPLY: finished
0114 02:54:23.946997 (+ 160us) log.cc:844] Serialized 29 byte log entry
0114 02:54:23.947114 (+ 117us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:23.947187 (+ 73us) write_op.cc:454] Released schema lock
0114 02:54:23.947214 (+ 27us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:23.957804 (+ 10590us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:23.957838 (+ 34us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2082,"apply.run_cpu_time_us":1272,"apply.run_wall_time_us":1272,"num_ops":2,"prepare.queue_time_us":25619,"prepare.run_cpu_time_us":1868,"prepare.run_wall_time_us":44322,"replication_time_us":29166}]]}
I20250114 02:54:25.376626 1153 heartbeater.cc:346] Connected to a master server at 127.0.107.126:45367
I20250114 02:54:25.348013 1965 ts_tablet_manager.cc:1918] T 0f1ea1608213492c9a216757896ff2a4 P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:25.379483 466 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_3139140b7eec4849ae715c295fbeaa6b" } new_table_name: "default.Table_69b77a6fd8ad45cc9e3dc32b202d93cd" modify_external_catalogs: true
I20250114 02:54:25.380640 1965 log.cc:1198] T 0f1ea1608213492c9a216757896ff2a4 P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/0f1ea1608213492c9a216757896ff2a4
I20250114 02:54:25.382440 1965 ts_tablet_manager.cc:1939] T 0f1ea1608213492c9a216757896ff2a4 P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:25.390264 957 raft_consensus.cc:2463] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 1.
I20250114 02:54:25.391227 932 tablet_service.cc:1514] Processing DeleteTablet for tablet 321b9ad6d1ac4d1c80c7a01faf2015af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:23 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:25.393723 789 tablet_service.cc:1514] Processing DeleteTablet for tablet 321b9ad6d1ac4d1c80c7a01faf2015af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:23 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:25.398187 1081 tablet_service.cc:1514] Processing DeleteTablet for tablet 321b9ad6d1ac4d1c80c7a01faf2015af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:23 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:25.402405 1990 tablet_replica.cc:331] T 321b9ad6d1ac4d1c80c7a01faf2015af P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:25.403363 1990 raft_consensus.cc:2238] T 321b9ad6d1ac4d1c80c7a01faf2015af P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:25.403836 1990 raft_consensus.cc:2267] T 321b9ad6d1ac4d1c80c7a01faf2015af P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:25.407779 1987 tablet.cc:1721] T 6448eea21ab2422c97da2494658ded3f P 525198593c8e47afaee54e74267b87e3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250114 02:54:25.412659 1990 ts_tablet_manager.cc:1905] T 321b9ad6d1ac4d1c80c7a01faf2015af P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:25.418474 1205 tablet_bootstrap.cc:492] T 84738a6c78b24c1fb64013048763b18f P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:25.418926 1205 ts_tablet_manager.cc:1397] T 84738a6c78b24c1fb64013048763b18f P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.596s user 0.016s sys 0.010s
I20250114 02:54:25.421262 1205 raft_consensus.cc:357] T 84738a6c78b24c1fb64013048763b18f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:25.422065 1205 raft_consensus.cc:383] T 84738a6c78b24c1fb64013048763b18f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:25.422322 1205 raft_consensus.cc:738] T 84738a6c78b24c1fb64013048763b18f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:25.422992 1206 tablet_bootstrap.cc:492] T 38fe8829d98547fb907ef78099df3d7f P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:25.423388 1206 ts_tablet_manager.cc:1397] T 38fe8829d98547fb907ef78099df3d7f P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.224s user 0.005s sys 0.004s
I20250114 02:54:25.425804 1206 raft_consensus.cc:357] T 38fe8829d98547fb907ef78099df3d7f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:25.426445 1206 raft_consensus.cc:383] T 38fe8829d98547fb907ef78099df3d7f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:25.426693 1206 raft_consensus.cc:738] T 38fe8829d98547fb907ef78099df3d7f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:25.427259 1206 consensus_queue.cc:260] T 38fe8829d98547fb907ef78099df3d7f P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
W20250114 02:54:25.438969 468 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.66:50663 (request call id 44) took 5146 ms (client timeout 1000 ms). Trace:
I20250114 02:54:25.440937 935 tablet_service.cc:1514] Processing DeleteTablet for tablet ba215eb8422d4f879dd0bf5996543962 with delete_type TABLET_DATA_DELETED (replaced by tablet f40c66314342453b898da3ac0e7e86bf) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:25.441589 941 tablet_service.cc:1514] Processing DeleteTablet for tablet 99b8acb1fe594bf3825c7b9121018db6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:16 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:25.443161 932 tablet_service.cc:1514] Processing DeleteTablet for tablet 290ad07ab8be4791a39194909e000843 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:13 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:25.443786 933 tablet_service.cc:1514] Processing DeleteTablet for tablet 9812754c610548d1b43084bbc94fc3e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:19 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:25.446111 465 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: default.Table_4fd891b63924429085d78f78cb177f0e [id=d36c71e7562b4ff1bc8be5cf2955000e] alter complete (version 1)
I20250114 02:54:25.450783 1206 ts_tablet_manager.cc:1428] T 38fe8829d98547fb907ef78099df3d7f P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.027s user 0.006s sys 0.001s
I20250114 02:54:25.450968 466 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:33540 (request call id 240) took 1410 ms. Trace:
I20250114 02:54:25.451637 1206 tablet_bootstrap.cc:492] T 6e858a8449444ef999b9bbef612249d3 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:25.451256 466 rpcz_store.cc:276] 0114 02:54:24.040260 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:25.376084 (+1335824us) service_pool.cc:225] Handling call
0114 02:54:25.381070 (+ 4986us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0114 02:54:25.384155 (+ 3085us) catalog_manager.cc:3627] Apply alter schema
0114 02:54:25.408275 (+ 24120us) catalog_manager.cc:3659] Acquired catalog manager lock
0114 02:54:25.408658 (+ 383us) catalog_manager.cc:3817] Updating metadata on disk
0114 02:54:25.448881 (+ 40223us) catalog_manager.cc:3848] Committing alterations to in-memory state
0114 02:54:25.450908 (+ 2027us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:25.418332 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:25.418699 (+ 367us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:25.418726 (+ 27us) write_op.cc:435] Acquired schema lock
0114 02:54:25.418740 (+ 14us) tablet.cc:659] Decoding operations
0114 02:54:25.418988 (+ 248us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:25.419023 (+ 35us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:25.419048 (+ 25us) tablet.cc:682] Acquiring locks for 1 operations
0114 02:54:25.419192 (+ 144us) tablet.cc:698] Row locks acquired
0114 02:54:25.419216 (+ 24us) write_op.cc:260] PREPARE: finished
0114 02:54:25.419317 (+ 101us) write_op.cc:270] Start()
0114 02:54:25.419393 (+ 76us) write_op.cc:276] Timestamp: P: 1736823265419218 usec, L: 0
0114 02:54:25.419410 (+ 17us) op_driver.cc:348] REPLICATION: starting
0114 02:54:25.419845 (+ 435us) log.cc:844] Serialized 405 byte log entry
0114 02:54:25.443649 (+ 23804us) op_driver.cc:464] REPLICATION: finished
0114 02:54:25.447634 (+ 3985us) write_op.cc:301] APPLY: starting
0114 02:54:25.447682 (+ 48us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:25.447770 (+ 88us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:25.447860 (+ 90us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:25.448105 (+ 245us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:25.448245 (+ 140us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:25.448294 (+ 49us) write_op.cc:312] APPLY: finished
0114 02:54:25.448480 (+ 186us) log.cc:844] Serialized 23 byte log entry
0114 02:54:25.448603 (+ 123us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:25.448662 (+ 59us) write_op.cc:454] Released schema lock
0114 02:54:25.448698 (+ 36us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":891,"child_traces":[["op",{"apply.queue_time_us":3838,"apply.run_cpu_time_us":1248,"apply.run_wall_time_us":1941,"num_ops":1,"prepare.queue_time_us":8461,"raft.queue_time_us":15900,"raft.run_cpu_time_us":46,"raft.run_wall_time_us":20,"replication_time_us":24161,"threads_started":2}]]}
I20250114 02:54:25.466302 1991 tablet.cc:1721] T 55b68586ef8b45439a7dd8dcd2efcd79 P 525198593c8e47afaee54e74267b87e3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250114 02:54:25.480046 789 tablet_service.cc:1514] Processing DeleteTablet for tablet 266da813b4514329be0ad97d07b7114f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:19 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:25.486513 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet 0f1ea1608213492c9a216757896ff2a4 (table default.Table_ae6b6fdfc1834cfb8089c860aedcc964 [id=7da003630a7a4e269d54fb2af8a3d4b5]) successfully deleted
W20250114 02:54:25.439319 468 rpcz_store.cc:269] 0114 02:54:20.292595 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:20.473652 (+181057us) service_pool.cc:225] Handling call
0114 02:54:20.564931 (+ 91279us) spinlock_profiling.cc:243] Waited 31.7 ms on lock 0x7b54000316d4. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616cad50 00007f646174df2a 00007f64633f2dec 00007f64634eadee 00007f6461248fc4 00007f6461248f52 00007f6461248e81 00007f6461248dfc 00007f64612480b2 00007f645f512dfc 00007f645f512236
0114 02:54:25.438756 (+4873825us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:23.144843 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:23.145323 (+ 480us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:23.145355 (+ 32us) write_op.cc:435] Acquired schema lock
0114 02:54:23.145375 (+ 20us) tablet.cc:659] Decoding operations
0114 02:54:23.145685 (+ 310us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:23.145739 (+ 54us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:23.145793 (+ 54us) tablet.cc:682] Acquiring locks for 1 operations
0114 02:54:23.145976 (+ 183us) tablet.cc:698] Row locks acquired
0114 02:54:23.145995 (+ 19us) write_op.cc:260] PREPARE: finished
0114 02:54:23.146125 (+ 130us) write_op.cc:270] Start()
0114 02:54:23.146236 (+ 111us) write_op.cc:276] Timestamp: P: 1736823263146037 usec, L: 0
0114 02:54:23.146258 (+ 22us) op_driver.cc:348] REPLICATION: starting
0114 02:54:23.195899 (+ 49641us) log.cc:844] Serialized 356 byte log entry
0114 02:54:23.197812 (+ 1913us) spinlock_profiling.cc:243] Waited 49 ms on lock 0x7b54000316d4. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616ca6d1 00007f6461741198 00007f6461bad3b1 00007f6461bacb88 00007f6461bb1821 00007f6461bb17d9 00007f6461bb1769 00007f6461bb1731 00007f6461bb095d 000000000059ea04 000000000059e839
0114 02:54:23.217632 (+ 19820us) op_driver.cc:464] REPLICATION: finished
0114 02:54:23.217903 (+ 271us) write_op.cc:301] APPLY: starting
0114 02:54:23.217976 (+ 73us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:23.218111 (+ 135us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:23.218139 (+ 28us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:23.218350 (+ 211us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:23.218523 (+ 173us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:23.218584 (+ 61us) write_op.cc:312] APPLY: finished
0114 02:54:23.218807 (+ 223us) log.cc:844] Serialized 23 byte log entry
0114 02:54:23.218975 (+ 168us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:23.219055 (+ 80us) write_op.cc:454] Released schema lock
0114 02:54:23.219101 (+ 46us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":54185,"reported_tablets":13,"spinlock_wait_cycles":72886144,"child_traces":[["op",{"apply.queue_time_us":98,"apply.run_cpu_time_us":1414,"apply.run_wall_time_us":1413,"num_ops":1,"prepare.queue_time_us":132,"prepare.run_cpu_time_us":4958,"prepare.run_wall_time_us":56327,"raft.queue_time_us":5946,"raft.run_cpu_time_us":25,"raft.run_wall_time_us":25,"replication_time_us":71249,"spinlock_wait_cycles":112683264}]]}
I20250114 02:54:25.497197 940 tablet_service.cc:1514] Processing DeleteTablet for tablet 66bfa92db9874750b3c340e31a06c412 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:16 UTC) from {username='slave'} at 127.0.0.1:55068
W20250114 02:54:25.497685 470 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.66:50663 (request call id 46) took 2871 ms (client timeout 1000 ms). Trace:
I20250114 02:54:25.497785 942 tablet_service.cc:1514] Processing DeleteTablet for tablet bc46815b4803418c95ea9c039a4fae88 with delete_type TABLET_DATA_DELETED (replaced by tablet 97f4e061d3b94ee9982410b7ce135453) from {username='slave'} at 127.0.0.1:55068
W20250114 02:54:25.497992 470 rpcz_store.cc:269] 0114 02:54:22.626239 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:23.041044 (+414805us) service_pool.cc:225] Handling call
0114 02:54:25.497489 (+2456445us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":86085,"reported_tablets":21}
I20250114 02:54:25.487141 1083 tablet_service.cc:1514] Processing DeleteTablet for tablet 53a8bb487ad64157ad3baecf5d866eca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:17 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:25.487545 1084 tablet_service.cc:1514] Processing DeleteTablet for tablet 266da813b4514329be0ad97d07b7114f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:19 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:25.490257 1081 tablet_service.cc:1514] Processing DeleteTablet for tablet 26328f8e32bc49ed9b224f0e08b9f949 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:21 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:25.491709 1086 tablet_service.cc:1514] Processing DeleteTablet for tablet 66bfa92db9874750b3c340e31a06c412 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:16 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:25.500960 942 tablet_service.cc:1514] Processing DeleteTablet for tablet ca986f81ed6848a89588366f3f400eec with delete_type TABLET_DATA_DELETED (replaced by tablet a2d68eccaaa64511a9ae36f2b07a7b59) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:25.500994 935 tablet_service.cc:1514] Processing DeleteTablet for tablet 26328f8e32bc49ed9b224f0e08b9f949 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:21 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:25.501487 941 tablet_service.cc:1514] Processing DeleteTablet for tablet 53a8bb487ad64157ad3baecf5d866eca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:17 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:25.495203 1961 raft_consensus.cc:491] T edc69b66bea54625a4b15867b75986b7 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:25.504521 796 tablet_service.cc:1514] Processing DeleteTablet for tablet ca986f81ed6848a89588366f3f400eec with delete_type TABLET_DATA_DELETED (replaced by tablet a2d68eccaaa64511a9ae36f2b07a7b59) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:25.504575 1961 raft_consensus.cc:513] T edc69b66bea54625a4b15867b75986b7 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:25.505409 796 tablet_service.cc:1514] Processing DeleteTablet for tablet 66bfa92db9874750b3c340e31a06c412 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:16 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:25.505785 797 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f1ea1608213492c9a216757896ff2a4 with delete_type TABLET_DATA_DELETED (replaced by tablet 550e84ac4d1b4ef58dfdea6e552ee940) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:25.506484 1961 leader_election.cc:290] T edc69b66bea54625a4b15867b75986b7 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:25.501825 1948 raft_consensus.cc:491] T 38fe8829d98547fb907ef78099df3d7f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:25.508245 1948 raft_consensus.cc:513] T 38fe8829d98547fb907ef78099df3d7f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:25.511476 1948 leader_election.cc:290] T 38fe8829d98547fb907ef78099df3d7f P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:25.511513 797 tablet_service.cc:1514] Processing DeleteTablet for tablet 9812754c610548d1b43084bbc94fc3e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:19 UTC) from {username='slave'} at 127.0.0.1:53078
W20250114 02:54:25.514848 471 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.65:49805 (request call id 48) took 2998 ms (client timeout 1000 ms). Trace:
I20250114 02:54:25.503459 1206 tablet_bootstrap.cc:654] T 6e858a8449444ef999b9bbef612249d3 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:25.520332 1102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc69b66bea54625a4b15867b75986b7" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:25.523092 1102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38fe8829d98547fb907ef78099df3d7f" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:25.524156 790 tablet_service.cc:1514] Processing DeleteTablet for tablet 26328f8e32bc49ed9b224f0e08b9f949 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:21 UTC) from {username='slave'} at 127.0.0.1:53078
W20250114 02:54:25.525055 753 leader_election.cc:343] T edc69b66bea54625a4b15867b75986b7 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:25.526427 935 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f1ea1608213492c9a216757896ff2a4 with delete_type TABLET_DATA_DELETED (replaced by tablet 550e84ac4d1b4ef58dfdea6e552ee940) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:25.526428 941 tablet_service.cc:1514] Processing DeleteTablet for tablet 46360f726fad4534a2daf67581346fc1 with delete_type TABLET_DATA_DELETED (replaced by tablet be7aad41ccdd4ef9bcfe8a985155708f) from {username='slave'} at 127.0.0.1:55068
W20250114 02:54:25.527781 753 leader_election.cc:343] T 38fe8829d98547fb907ef78099df3d7f P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:25.529467 1086 tablet_service.cc:1514] Processing DeleteTablet for tablet bc46815b4803418c95ea9c039a4fae88 with delete_type TABLET_DATA_DELETED (replaced by tablet 97f4e061d3b94ee9982410b7ce135453) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:25.530092 1083 tablet_service.cc:1514] Processing DeleteTablet for tablet bc46815b4803418c95ea9c039a4fae88 with delete_type TABLET_DATA_DELETED (replaced by tablet 97f4e061d3b94ee9982410b7ce135453) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:25.533632 1084 tablet_service.cc:1514] Processing DeleteTablet for tablet ca986f81ed6848a89588366f3f400eec with delete_type TABLET_DATA_DELETED (replaced by tablet a2d68eccaaa64511a9ae36f2b07a7b59) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:25.517266 468 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_d101e66d79a64b6f9233687a3cb522a3" } new_table_name: "default.Table_b4dc2961391c43fcb08fa062d4e45b3c" modify_external_catalogs: true
W20250114 02:54:25.519520 469 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.65:49805 (request call id 49) took 1990 ms (client timeout 1000 ms). Trace:
I20250114 02:54:25.524413 470 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_08599872e48848e39e5bd7bcbfcec783"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:25.540205 1957 tablet.cc:1721] T 0304df55431e42e1a999ed17dc35e6c0 P 90d6c2c58bb54e0eafb365625ac33871: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250114 02:54:25.526804 465 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.66:50663 (request call id 47) took 1773 ms (client timeout 1000 ms). Trace:
W20250114 02:54:25.528491 463 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.67:34825 (request call id 44) took 3246 ms (client timeout 1000 ms). Trace:
I20250114 02:54:25.544548 1741 raft_consensus.cc:491] T 92cd8eccacf34ab7a66b3bbc69df3697 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:25.545392 1741 raft_consensus.cc:513] T 92cd8eccacf34ab7a66b3bbc69df3697 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:25.547591 1741 leader_election.cc:290] T 92cd8eccacf34ab7a66b3bbc69df3697 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:25.560786 1207 tablet_bootstrap.cc:654] T e15e6e52d7294aeeb4463b3a76022ec3 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:25.486708 1992 tablet_replica.cc:331] T 321b9ad6d1ac4d1c80c7a01faf2015af P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
W20250114 02:54:25.564283 465 rpcz_store.cc:269] 0114 02:54:23.752930 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:23.767378 (+ 14448us) service_pool.cc:225] Handling call
0114 02:54:23.890393 (+123015us) spinlock_profiling.cc:243] Waited 74.4 ms on lock 0x7b54000316d4. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616cad50 00007f646174df2a 00007f64633f2dec 00007f64634eadee 00007f6461248fc4 00007f6461248f52 00007f6461248e81 00007f6461248dfc 00007f64612480b2 00007f645f512dfc 00007f645f512236
0114 02:54:24.806796 (+916403us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:24.806823 (+ 27us) write_op.cc:454] Released schema lock
0114 02:54:25.526740 (+719917us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:24.769868 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:24.770204 (+ 336us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:24.770229 (+ 25us) write_op.cc:435] Acquired schema lock
0114 02:54:24.770240 (+ 11us) tablet.cc:659] Decoding operations
0114 02:54:24.770543 (+ 303us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:24.770582 (+ 39us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:24.770605 (+ 23us) tablet.cc:682] Acquiring locks for 2 operations
0114 02:54:24.770841 (+ 236us) tablet.cc:698] Row locks acquired
0114 02:54:24.770855 (+ 14us) write_op.cc:260] PREPARE: finished
0114 02:54:24.770959 (+ 104us) write_op.cc:270] Start()
0114 02:54:24.771033 (+ 74us) write_op.cc:276] Timestamp: P: 1736823264770924 usec, L: 0
0114 02:54:24.771049 (+ 16us) op_driver.cc:348] REPLICATION: starting
0114 02:54:24.771501 (+ 452us) log.cc:844] Serialized 907 byte log entry
0114 02:54:24.782986 (+ 11485us) op_driver.cc:464] REPLICATION: finished
0114 02:54:24.783297 (+ 311us) write_op.cc:301] APPLY: starting
0114 02:54:24.783346 (+ 49us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:24.783463 (+ 117us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:24.783545 (+ 82us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:24.783782 (+ 237us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:24.783917 (+ 135us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0114 02:54:24.783957 (+ 40us) write_op.cc:312] APPLY: finished
0114 02:54:24.784139 (+ 182us) log.cc:844] Serialized 29 byte log entry
0114 02:54:24.784318 (+ 179us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:24.784418 (+ 100us) write_op.cc:454] Released schema lock
0114 02:54:24.784461 (+ 43us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0114 02:54:25.415564 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:25.416034 (+ 470us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:25.416069 (+ 35us) write_op.cc:435] Acquired schema lock
0114 02:54:25.416082 (+ 13us) tablet.cc:659] Decoding operations
0114 02:54:25.416451 (+ 369us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:25.416507 (+ 56us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:25.416531 (+ 24us) tablet.cc:682] Acquiring locks for 1 operations
0114 02:54:25.416753 (+ 222us) tablet.cc:698] Row locks acquired
0114 02:54:25.416765 (+ 12us) write_op.cc:260] PREPARE: finished
0114 02:54:25.416926 (+ 161us) write_op.cc:270] Start()
0114 02:54:25.417018 (+ 92us) write_op.cc:276] Timestamp: P: 1736823265416786 usec, L: 0
0114 02:54:25.417043 (+ 25us) op_driver.cc:348] REPLICATION: starting
0114 02:54:25.417775 (+ 732us) log.cc:844] Serialized 356 byte log entry
0114 02:54:25.443286 (+ 25511us) op_driver.cc:464] REPLICATION: finished
0114 02:54:25.444367 (+ 1081us) write_op.cc:301] APPLY: starting
0114 02:54:25.444445 (+ 78us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:25.444561 (+ 116us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:25.444683 (+ 122us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:25.444926 (+ 243us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:25.445094 (+ 168us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:25.445151 (+ 57us) write_op.cc:312] APPLY: finished
0114 02:54:25.445394 (+ 243us) log.cc:844] Serialized 23 byte log entry
0114 02:54:25.445565 (+ 171us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:25.445631 (+ 66us) write_op.cc:454] Released schema lock
0114 02:54:25.445672 (+ 41us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:25.447345 (+ 1673us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:25.447372 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":20234,"reported_tablets":23,"spinlock_wait_cycles":171172224,"child_traces":[["op",{"apply.queue_time_us":177,"apply.run_cpu_time_us":1361,"apply.run_wall_time_us":1361,"num_ops":2,"prepare.queue_time_us":92,"prepare.run_cpu_time_us":1878,"prepare.run_wall_time_us":1878,"raft.queue_time_us":1195,"raft.run_cpu_time_us":756,"raft.run_wall_time_us":756,"replication_time_us":11849}],["op",{"apply.queue_time_us":771,"apply.run_cpu_time_us":1591,"apply.run_wall_time_us":2962,"num_ops":1,"prepare.queue_time_us":21286,"prepare.run_cpu_time_us":2666,"prepare.run_wall_time_us":2667,"raft.queue_time_us":17179,"raft.run_cpu_time_us":1149,"raft.run_wall_time_us":1149,"replication_time_us":26104}]]}
W20250114 02:54:25.571959 470 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_08599872e48848e39e5bd7bcbfcec783 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:25.576726 1992 raft_consensus.cc:2238] T 321b9ad6d1ac4d1c80c7a01faf2015af P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:25.581972 1965 tablet_replica.cc:331] T 321b9ad6d1ac4d1c80c7a01faf2015af P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:25.586534 1965 raft_consensus.cc:2238] T 321b9ad6d1ac4d1c80c7a01faf2015af P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Raft consensus shutting down.
W20250114 02:54:25.585137 469 rpcz_store.cc:269] 0114 02:54:23.528152 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:23.730135 (+201983us) service_pool.cc:225] Handling call
0114 02:54:23.786805 (+ 56670us) spinlock_profiling.cc:243] Waited 51.9 ms on lock 0x7b54000316d4. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616cad50 00007f646174df2a 00007f64633f2dec 00007f64634eadee 00007f6461248fc4 00007f6461248f52 00007f6461248e81 00007f6461248dfc 00007f64612480b2 00007f645f512dfc 00007f645f512236
0114 02:54:25.518356 (+1731551us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:24.619455 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:24.619771 (+ 316us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:24.619793 (+ 22us) write_op.cc:435] Acquired schema lock
0114 02:54:24.619806 (+ 13us) tablet.cc:659] Decoding operations
0114 02:54:24.620091 (+ 285us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:24.620130 (+ 39us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:24.620151 (+ 21us) tablet.cc:682] Acquiring locks for 2 operations
0114 02:54:24.620410 (+ 259us) tablet.cc:698] Row locks acquired
0114 02:54:24.620421 (+ 11us) write_op.cc:260] PREPARE: finished
0114 02:54:24.620521 (+ 100us) write_op.cc:270] Start()
0114 02:54:24.620594 (+ 73us) write_op.cc:276] Timestamp: P: 1736823264620457 usec, L: 0
0114 02:54:24.620608 (+ 14us) op_driver.cc:348] REPLICATION: starting
0114 02:54:24.621020 (+ 412us) log.cc:844] Serialized 907 byte log entry
0114 02:54:24.631874 (+ 10854us) op_driver.cc:464] REPLICATION: finished
0114 02:54:24.637242 (+ 5368us) write_op.cc:301] APPLY: starting
0114 02:54:24.637299 (+ 57us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:24.637426 (+ 127us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:24.637521 (+ 95us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:24.637748 (+ 227us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:24.637875 (+ 127us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0114 02:54:24.637912 (+ 37us) write_op.cc:312] APPLY: finished
0114 02:54:24.638086 (+ 174us) log.cc:844] Serialized 29 byte log entry
0114 02:54:24.638681 (+ 595us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:24.638750 (+ 69us) write_op.cc:454] Released schema lock
0114 02:54:24.638780 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:24.638979 (+ 199us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:24.639001 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":57669,"reported_tablets":18,"spinlock_wait_cycles":119393664,"child_traces":[["op",{"apply.queue_time_us":5209,"apply.run_cpu_time_us":1268,"apply.run_wall_time_us":1716,"num_ops":2,"prepare.queue_time_us":94,"prepare.run_cpu_time_us":1803,"prepare.run_wall_time_us":1803,"raft.queue_time_us":150,"raft.run_cpu_time_us":622,"raft.run_wall_time_us":622,"replication_time_us":11177}]]}
W20250114 02:54:25.557351 467 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.67:34825 (request call id 45) took 2229 ms (client timeout 1000 ms). Trace:
W20250114 02:54:25.592525 467 rpcz_store.cc:269] 0114 02:54:23.327672 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:23.361932 (+ 34260us) service_pool.cc:225] Handling call
0114 02:54:23.496641 (+134709us) spinlock_profiling.cc:243] Waited 56.2 ms on lock 0x7b54000316d4. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616cad50 00007f646174df2a 00007f64633f2dec 00007f64634eadee 00007f6461248fc4 00007f6461248f52 00007f6461248e81 00007f6461248dfc 00007f64612480b2 00007f645f512dfc 00007f645f512236
0114 02:54:25.557278 (+2060637us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:24.138546 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:24.138858 (+ 312us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:24.138882 (+ 24us) write_op.cc:435] Acquired schema lock
0114 02:54:24.138893 (+ 11us) tablet.cc:659] Decoding operations
0114 02:54:24.139140 (+ 247us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:24.139177 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:24.139198 (+ 21us) tablet.cc:682] Acquiring locks for 1 operations
0114 02:54:24.139328 (+ 130us) tablet.cc:698] Row locks acquired
0114 02:54:24.139340 (+ 12us) write_op.cc:260] PREPARE: finished
0114 02:54:24.139444 (+ 104us) write_op.cc:270] Start()
0114 02:54:24.139518 (+ 74us) write_op.cc:276] Timestamp: P: 1736823264139369 usec, L: 0
0114 02:54:24.139532 (+ 14us) op_driver.cc:348] REPLICATION: starting
0114 02:54:24.139973 (+ 441us) log.cc:844] Serialized 530 byte log entry
0114 02:54:24.148913 (+ 8940us) op_driver.cc:464] REPLICATION: finished
0114 02:54:24.163943 (+ 15030us) write_op.cc:301] APPLY: starting
0114 02:54:24.164015 (+ 72us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:24.164124 (+ 109us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:24.173217 (+ 9093us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:24.173412 (+ 195us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:24.173542 (+ 130us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:24.173583 (+ 41us) write_op.cc:312] APPLY: finished
0114 02:54:24.173760 (+ 177us) log.cc:844] Serialized 23 byte log entry
0114 02:54:24.173935 (+ 175us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:24.174015 (+ 80us) write_op.cc:454] Released schema lock
0114 02:54:24.174056 (+ 41us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:24.178169 (+ 4113us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:24.178201 (+ 32us) write_op.cc:454] Released schema lock
Related trace 'op':
0114 02:54:24.178075 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:24.178428 (+ 353us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:24.178449 (+ 21us) write_op.cc:435] Acquired schema lock
0114 02:54:24.178466 (+ 17us) tablet.cc:659] Decoding operations
0114 02:54:24.178691 (+ 225us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:24.178727 (+ 36us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:24.178748 (+ 21us) tablet.cc:682] Acquiring locks for 1 operations
0114 02:54:24.178878 (+ 130us) tablet.cc:698] Row locks acquired
0114 02:54:24.178892 (+ 14us) write_op.cc:260] PREPARE: finished
0114 02:54:24.178998 (+ 106us) write_op.cc:270] Start()
0114 02:54:24.179073 (+ 75us) write_op.cc:276] Timestamp: P: 1736823264178918 usec, L: 0
0114 02:54:24.179087 (+ 14us) op_driver.cc:348] REPLICATION: starting
0114 02:54:24.179535 (+ 448us) log.cc:844] Serialized 356 byte log entry
0114 02:54:24.227141 (+ 47606us) op_driver.cc:464] REPLICATION: finished
0114 02:54:24.228701 (+ 1560us) write_op.cc:301] APPLY: starting
0114 02:54:24.228776 (+ 75us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:24.228900 (+ 124us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:24.229046 (+ 146us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:24.229234 (+ 188us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:24.229421 (+ 187us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:24.229481 (+ 60us) write_op.cc:312] APPLY: finished
0114 02:54:24.229691 (+ 210us) log.cc:844] Serialized 23 byte log entry
0114 02:54:24.229858 (+ 167us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:24.229933 (+ 75us) write_op.cc:454] Released schema lock
0114 02:54:24.229971 (+ 38us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:24.230201 (+ 230us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:24.230235 (+ 34us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":19550,"reported_tablets":27,"spinlock_wait_cycles":129373568,"child_traces":[["op",{"apply.queue_time_us":14858,"apply.run_cpu_time_us":1388,"apply.run_wall_time_us":10324,"mutex_wait_us":290,"num_ops":1,"prepare.queue_time_us":12178,"prepare.run_cpu_time_us":1786,"prepare.run_wall_time_us":2060,"raft.queue_time_us":557,"raft.run_cpu_time_us":674,"raft.run_wall_time_us":675,"replication_time_us":9287}],["op",{"apply.queue_time_us":1370,"apply.run_cpu_time_us":1474,"apply.run_wall_time_us":1475,"mutex_wait_us":2461,"num_ops":1,"prepare.queue_time_us":105,"prepare.run_cpu_time_us":1772,"prepare.run_wall_time_us":2431,"raft.queue_time_us":2254,"raft.run_cpu_time_us":589,"raft.run_wall_time_us":588,"replication_time_us":47944}]]}
I20250114 02:54:25.577565 1958 tablet.cc:1721] T 0304df55431e42e1a999ed17dc35e6c0 P 819f9e3412e54d44abb11d8b3ca877c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250114 02:54:25.584199 1741 consensus_queue.cc:1035] T d3fb28717c6f4844af84bb817fe7dfc4 P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:25.594996 1948 consensus_queue.cc:1035] T f2b6812e8c1d44529d0284be490df648 P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:25.599395 751 leader_election.cc:304] T ce7d6088fc884a248918d2348adcaccf P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters: 90d6c2c58bb54e0eafb365625ac33871
W20250114 02:54:25.607051 464 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.67:34825 (request call id 46) took 1258 ms (client timeout 1000 ms). Trace:
I20250114 02:54:25.608471 1948 raft_consensus.cc:2758] T ce7d6088fc884a248918d2348adcaccf P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
W20250114 02:54:25.536316 471 rpcz_store.cc:269] 0114 02:54:22.516709 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:22.748051 (+231342us) service_pool.cc:225] Handling call
0114 02:54:25.514780 (+2766729us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:23.336881 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:23.337351 (+ 470us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:23.337383 (+ 32us) write_op.cc:435] Acquired schema lock
0114 02:54:23.337400 (+ 17us) tablet.cc:659] Decoding operations
0114 02:54:23.337739 (+ 339us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:23.337793 (+ 54us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:23.337824 (+ 31us) tablet.cc:682] Acquiring locks for 1 operations
0114 02:54:23.338004 (+ 180us) tablet.cc:698] Row locks acquired
0114 02:54:23.338022 (+ 18us) write_op.cc:260] PREPARE: finished
0114 02:54:23.338436 (+ 414us) write_op.cc:270] Start()
0114 02:54:23.338544 (+ 108us) write_op.cc:276] Timestamp: P: 1736823263338379 usec, L: 0
0114 02:54:23.338565 (+ 21us) op_driver.cc:348] REPLICATION: starting
0114 02:54:23.426865 (+ 88300us) spinlock_profiling.cc:243] Waited 86.4 ms on lock 0x7b5400032198. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64624790f4 00007f6461719389 00007f64616d51f1 00007f64616d4eca 00007f6461740cd5 00007f6461741179 00007f6461bad3b1 00007f6461bacb88 00007f6461bb1821 00007f6461bb17d9 00007f6461bb1769
0114 02:54:23.427186 (+ 321us) log.cc:844] Serialized 356 byte log entry
0114 02:54:23.440101 (+ 12915us) op_driver.cc:464] REPLICATION: finished
0114 02:54:23.446762 (+ 6661us) write_op.cc:301] APPLY: starting
0114 02:54:23.446834 (+ 72us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:23.446957 (+ 123us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:23.446985 (+ 28us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:23.447186 (+ 201us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:23.447356 (+ 170us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:23.447416 (+ 60us) write_op.cc:312] APPLY: finished
0114 02:54:23.447639 (+ 223us) log.cc:844] Serialized 23 byte log entry
0114 02:54:23.447807 (+ 168us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:23.447890 (+ 83us) write_op.cc:454] Released schema lock
0114 02:54:23.447929 (+ 39us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:23.460258 (+ 12329us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:23.460295 (+ 37us) write_op.cc:454] Released schema lock
0114 02:54:23.504637 (+ 44342us) spinlock_profiling.cc:243] Waited 74.6 ms on lock 0x7b58000d01b0. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616cad50 00007f64616bfa94 00007f64616c49d1 00007f64616c4959 00007f64616c48e9 00007f64616c48b1 00007f64616c3a7d 000000000059ea04 000000000059e839 00007f645a7fc866 00007f645a7ffcc1
Metrics: {"mutex_wait_us":40130,"reported_tablets":18,"child_traces":[["op",{"apply.queue_time_us":6464,"apply.run_cpu_time_us":1418,"apply.run_wall_time_us":1419,"num_ops":1,"prepare.queue_time_us":127,"prepare.run_cpu_time_us":11238,"prepare.run_wall_time_us":123359,"raft.queue_time_us":5975,"raft.run_cpu_time_us":2190,"raft.run_wall_time_us":75287,"replication_time_us":101418,"spinlock_wait_cycles":370174080,"thread_start_us":32493,"threads_started":1}]]}
I20250114 02:54:25.609071 1948 consensus_queue.cc:1035] T ca986f81ed6848a89588366f3f400eec P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250114 02:54:25.575331 463 rpcz_store.cc:269] 0114 02:54:22.282073 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:22.473744 (+191671us) service_pool.cc:225] Handling call
0114 02:54:25.528319 (+3054575us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:23.047507 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:23.047842 (+ 335us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:23.047866 (+ 24us) write_op.cc:435] Acquired schema lock
0114 02:54:23.047880 (+ 14us) tablet.cc:659] Decoding operations
0114 02:54:23.048156 (+ 276us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:23.048196 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:23.048237 (+ 41us) tablet.cc:682] Acquiring locks for 1 operations
0114 02:54:23.048374 (+ 137us) tablet.cc:698] Row locks acquired
0114 02:54:23.048395 (+ 21us) write_op.cc:260] PREPARE: finished
0114 02:54:23.048502 (+ 107us) write_op.cc:270] Start()
0114 02:54:23.048582 (+ 80us) write_op.cc:276] Timestamp: P: 1736823263048432 usec, L: 0
0114 02:54:23.048597 (+ 15us) op_driver.cc:348] REPLICATION: starting
0114 02:54:23.049043 (+ 446us) log.cc:844] Serialized 530 byte log entry
0114 02:54:23.060250 (+ 11207us) op_driver.cc:464] REPLICATION: finished
0114 02:54:23.060813 (+ 563us) write_op.cc:301] APPLY: starting
0114 02:54:23.060865 (+ 52us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:23.060952 (+ 87us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:23.061040 (+ 88us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:23.061208 (+ 168us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:23.061338 (+ 130us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:23.061379 (+ 41us) write_op.cc:312] APPLY: finished
0114 02:54:23.061577 (+ 198us) log.cc:844] Serialized 23 byte log entry
0114 02:54:23.061718 (+ 141us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:23.061775 (+ 57us) write_op.cc:454] Released schema lock
0114 02:54:23.061803 (+ 28us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:23.062028 (+ 225us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:23.062052 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":44990,"reported_tablets":24,"child_traces":[["op",{"apply.queue_time_us":378,"apply.run_cpu_time_us":1179,"apply.run_wall_time_us":1179,"num_ops":1,"prepare.queue_time_us":108,"prepare.run_cpu_time_us":1826,"prepare.run_wall_time_us":1826,"raft.queue_time_us":2426,"raft.run_cpu_time_us":626,"raft.run_wall_time_us":626,"replication_time_us":11518}]]}
I20250114 02:54:25.615746 1992 raft_consensus.cc:2267] T 321b9ad6d1ac4d1c80c7a01faf2015af P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:25.617848 1992 ts_tablet_manager.cc:1905] T 321b9ad6d1ac4d1c80c7a01faf2015af P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:25.620820 789 tablet_service.cc:1514] Processing DeleteTablet for tablet f0bd9bd5a17d418d8082ed0ebd1bf4cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:19 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:25.600585 817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce7d6088fc884a248918d2348adcaccf" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:25.624706 817 raft_consensus.cc:2388] T ce7d6088fc884a248918d2348adcaccf P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 525198593c8e47afaee54e74267b87e3 in current term 1: Already voted for candidate 90d6c2c58bb54e0eafb365625ac33871 in this term.
I20250114 02:54:25.625305 471 master_service.cc:946] ReplaceTablet: received request to replace tablet 1381e26725174b9c877075cf5444f566 from {username='slave'} at 127.0.0.1:33540
I20250114 02:54:25.631287 1965 raft_consensus.cc:2267] T 321b9ad6d1ac4d1c80c7a01faf2015af P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250114 02:54:25.607517 464 rpcz_store.cc:269] 0114 02:54:24.348311 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:24.376100 (+ 27789us) service_pool.cc:225] Handling call
0114 02:54:25.238028 (+861928us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:25.238055 (+ 27us) write_op.cc:454] Released schema lock
0114 02:54:25.606793 (+368738us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:25.220755 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:25.221102 (+ 347us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:25.221125 (+ 23us) write_op.cc:435] Acquired schema lock
0114 02:54:25.221137 (+ 12us) tablet.cc:659] Decoding operations
0114 02:54:25.221392 (+ 255us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:25.221460 (+ 68us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:25.221482 (+ 22us) tablet.cc:682] Acquiring locks for 1 operations
0114 02:54:25.221620 (+ 138us) tablet.cc:698] Row locks acquired
0114 02:54:25.221632 (+ 12us) write_op.cc:260] PREPARE: finished
0114 02:54:25.221744 (+ 112us) write_op.cc:270] Start()
0114 02:54:25.221824 (+ 80us) write_op.cc:276] Timestamp: P: 1736823265221661 usec, L: 0
0114 02:54:25.221839 (+ 15us) op_driver.cc:348] REPLICATION: starting
0114 02:54:25.222293 (+ 454us) log.cc:844] Serialized 530 byte log entry
0114 02:54:25.230437 (+ 8144us) op_driver.cc:464] REPLICATION: finished
0114 02:54:25.230777 (+ 340us) write_op.cc:301] APPLY: starting
0114 02:54:25.230829 (+ 52us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:25.230915 (+ 86us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:25.230996 (+ 81us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:25.231136 (+ 140us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:25.231257 (+ 121us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:25.231297 (+ 40us) write_op.cc:312] APPLY: finished
0114 02:54:25.231458 (+ 161us) log.cc:844] Serialized 23 byte log entry
0114 02:54:25.231580 (+ 122us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:25.231640 (+ 60us) write_op.cc:454] Released schema lock
0114 02:54:25.231668 (+ 28us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":6765,"reported_tablets":29,"child_traces":[["op",{"apply.queue_time_us":190,"apply.run_cpu_time_us":1049,"apply.run_wall_time_us":1049,"num_ops":1,"prepare.queue_time_us":3422,"prepare.run_cpu_time_us":1841,"prepare.run_wall_time_us":1840,"raft.queue_time_us":2431,"raft.run_cpu_time_us":643,"raft.run_wall_time_us":642,"replication_time_us":8509}]]}
I20250114 02:54:25.635237 1965 ts_tablet_manager.cc:1905] T 321b9ad6d1ac4d1c80c7a01faf2015af P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:25.636940 1741 consensus_queue.cc:1035] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:54:25.561691 1084 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f1ea1608213492c9a216757896ff2a4 with delete_type TABLET_DATA_DELETED (replaced by tablet 550e84ac4d1b4ef58dfdea6e552ee940) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:25.562479 1081 tablet_service.cc:1514] Processing DeleteTablet for tablet 46360f726fad4534a2daf67581346fc1 with delete_type TABLET_DATA_DELETED (replaced by tablet be7aad41ccdd4ef9bcfe8a985155708f) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:25.603564 1101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce7d6088fc884a248918d2348adcaccf" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:25.616519 1083 tablet_service.cc:1514] Processing DeleteTablet for tablet f0bd9bd5a17d418d8082ed0ebd1bf4cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:19 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:25.611454 1102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92cd8eccacf34ab7a66b3bbc69df3697" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:25.650758 789 tablet_service.cc:1514] Processing DeleteTablet for tablet 46360f726fad4534a2daf67581346fc1 with delete_type TABLET_DATA_DELETED (replaced by tablet be7aad41ccdd4ef9bcfe8a985155708f) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:25.611496 1990 ts_tablet_manager.cc:1918] T 321b9ad6d1ac4d1c80c7a01faf2015af P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:25.652264 1990 log.cc:1198] T 321b9ad6d1ac4d1c80c7a01faf2015af P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/321b9ad6d1ac4d1c80c7a01faf2015af
I20250114 02:54:25.653076 1990 ts_tablet_manager.cc:1939] T 321b9ad6d1ac4d1c80c7a01faf2015af P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
W20250114 02:54:25.652968 753 leader_election.cc:343] T 92cd8eccacf34ab7a66b3bbc69df3697 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:25.657406 464 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_b8b1e6d9bd0742f7880a7f55e986d0ee" } modify_external_catalogs: true
I20250114 02:54:25.657940 464 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_b8b1e6d9bd0742f7880a7f55e986d0ee" } modify_external_catalogs: true
I20250114 02:54:25.663064 817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42cee9e8e26147b593ed73fca6ac1183" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:25.663739 817 raft_consensus.cc:3054] T 42cee9e8e26147b593ed73fca6ac1183 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:25.664053 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet 321b9ad6d1ac4d1c80c7a01faf2015af (table default.Table_efc3b170c00f4ce09b2601f7f5ebf3b8 [id=dba5ff68f085474caeaf34c387ecac6f]) successfully deleted
I20250114 02:54:25.664422 1841 raft_consensus.cc:491] T ddfbab181b1f474192c664cd0316037f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 02:54:25.664903 458 catalog_manager.cc:4839] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 53a8bb487ad64157ad3baecf5d866eca because the tablet was not found. No further retry: Not found: Tablet not found: 53a8bb487ad64157ad3baecf5d866eca
I20250114 02:54:25.664860 1841 raft_consensus.cc:513] T ddfbab181b1f474192c664cd0316037f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:25.666782 1841 leader_election.cc:290] T ddfbab181b1f474192c664cd0316037f P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:25.672188 1099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42cee9e8e26147b593ed73fca6ac1183" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:25.672818 1099 raft_consensus.cc:3054] T 42cee9e8e26147b593ed73fca6ac1183 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:25.672948 1990 tablet_replica.cc:331] T 266da813b4514329be0ad97d07b7114f P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
W20250114 02:54:25.687218 462 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.65:49805 (request call id 50) took 1142 ms (client timeout 1000 ms). Trace:
W20250114 02:54:25.689010 462 rpcz_store.cc:269] 0114 02:54:24.544583 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:24.670321 (+125738us) service_pool.cc:225] Handling call
0114 02:54:25.686988 (+1016667us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":21}
I20250114 02:54:25.641929 1987 tablet.cc:1721] T 0304df55431e42e1a999ed17dc35e6c0 P 525198593c8e47afaee54e74267b87e3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250114 02:54:25.706259 959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddfbab181b1f474192c664cd0316037f" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:25.707378 959 raft_consensus.cc:2463] T ddfbab181b1f474192c664cd0316037f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 0.
I20250114 02:54:25.643639 1205 consensus_queue.cc:260] T 84738a6c78b24c1fb64013048763b18f P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:25.717707 1040 leader_election.cc:304] T ddfbab181b1f474192c664cd0316037f P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:25.718472 1841 raft_consensus.cc:2798] T ddfbab181b1f474192c664cd0316037f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:25.718870 1841 raft_consensus.cc:491] T ddfbab181b1f474192c664cd0316037f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:25.719241 1841 raft_consensus.cc:3054] T ddfbab181b1f474192c664cd0316037f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:25.727066 1841 raft_consensus.cc:513] T ddfbab181b1f474192c664cd0316037f P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
W20250114 02:54:25.729004 468 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5d545ac27a7f49faace0655713405cea consensus state has no leader
I20250114 02:54:25.729025 1841 leader_election.cc:290] T ddfbab181b1f474192c664cd0316037f P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Requested vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:25.729550 468 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:33540 (request call id 241) took 1637 ms. Trace:
I20250114 02:54:25.730325 959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddfbab181b1f474192c664cd0316037f" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
I20250114 02:54:25.731070 959 raft_consensus.cc:3054] T ddfbab181b1f474192c664cd0316037f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:25.734426 898 leader_election.cc:304] T 46360f726fad4534a2daf67581346fc1 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 525198593c8e47afaee54e74267b87e3; no voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:25.735144 1944 raft_consensus.cc:2743] T 46360f726fad4534a2daf67581346fc1 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:25.737859 1205 ts_tablet_manager.cc:1428] T 84738a6c78b24c1fb64013048763b18f P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.319s user 0.005s sys 0.000s
I20250114 02:54:25.739289 1205 tablet_bootstrap.cc:492] T 92cd8eccacf34ab7a66b3bbc69df3697 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:25.741675 467 master_service.cc:946] ReplaceTablet: received request to replace tablet 5ac7708550954a209e2a8d217c3b39bd from {username='slave'} at 127.0.0.1:33540
I20250114 02:54:25.729856 468 rpcz_store.cc:276] 0114 02:54:24.092547 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:25.489181 (+1396634us) service_pool.cc:225] Handling call
0114 02:54:25.517159 (+ 27978us) spinlock_profiling.cc:243] Waited 27 ms on lock 0x7b54000316d4. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616cad50 00007f646174df2a 00007f64633f2dec 00007f64634ec986 00007f6461259644 00007f64612595d2 00007f6461259501 00007f646125947c 00007f6461258732 00007f645f512dfc 00007f645f512236
0114 02:54:25.562065 (+ 44906us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0114 02:54:25.562737 (+ 672us) catalog_manager.cc:3627] Apply alter schema
0114 02:54:25.563359 (+ 622us) catalog_manager.cc:3659] Acquired catalog manager lock
0114 02:54:25.563587 (+ 228us) catalog_manager.cc:3817] Updating metadata on disk
0114 02:54:25.686784 (+123197us) catalog_manager.cc:3848] Committing alterations to in-memory state
0114 02:54:25.729493 (+ 42709us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:25.610146 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:25.610518 (+ 372us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:25.610543 (+ 25us) write_op.cc:435] Acquired schema lock
0114 02:54:25.610557 (+ 14us) tablet.cc:659] Decoding operations
0114 02:54:25.610787 (+ 230us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:25.610825 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:25.610852 (+ 27us) tablet.cc:682] Acquiring locks for 1 operations
0114 02:54:25.611004 (+ 152us) tablet.cc:698] Row locks acquired
0114 02:54:25.611022 (+ 18us) write_op.cc:260] PREPARE: finished
0114 02:54:25.611133 (+ 111us) write_op.cc:270] Start()
0114 02:54:25.611221 (+ 88us) write_op.cc:276] Timestamp: P: 1736823265611011 usec, L: 0
0114 02:54:25.611234 (+ 13us) op_driver.cc:348] REPLICATION: starting
0114 02:54:25.611710 (+ 476us) log.cc:844] Serialized 405 byte log entry
0114 02:54:25.666417 (+ 54707us) op_driver.cc:464] REPLICATION: finished
0114 02:54:25.666922 (+ 505us) write_op.cc:301] APPLY: starting
0114 02:54:25.666981 (+ 59us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:25.667088 (+ 107us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:25.667215 (+ 127us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:25.667423 (+ 208us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:25.667586 (+ 163us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:25.667638 (+ 52us) write_op.cc:312] APPLY: finished
0114 02:54:25.667833 (+ 195us) log.cc:844] Serialized 23 byte log entry
0114 02:54:25.667965 (+ 132us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:25.668039 (+ 74us) write_op.cc:454] Released schema lock
0114 02:54:25.673774 (+ 5735us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:25.674047 (+ 273us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:25.674071 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":62173056,"child_traces":[["op",{"apply.queue_time_us":314,"apply.run_cpu_time_us":1415,"apply.run_wall_time_us":7098,"num_ops":1,"prepare.queue_time_us":90,"prepare.run_cpu_time_us":1854,"prepare.run_wall_time_us":1854,"raft.queue_time_us":1720,"raft.run_cpu_time_us":571,"raft.run_wall_time_us":571,"replication_time_us":55071}]]}
I20250114 02:54:25.749074 1990 raft_consensus.cc:2238] T 266da813b4514329be0ad97d07b7114f P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Raft consensus shutting down.
W20250114 02:54:25.753473 751 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.66:44037, user_credentials={real_user=slave}} blocked reactor thread for 100333us
I20250114 02:54:25.753747 1990 pending_rounds.cc:62] T 266da813b4514329be0ad97d07b7114f P 90d6c2c58bb54e0eafb365625ac33871: Trying to abort 1 pending ops.
I20250114 02:54:25.754056 1990 pending_rounds.cc:69] T 266da813b4514329be0ad97d07b7114f P 90d6c2c58bb54e0eafb365625ac33871: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114028083378245632 op_type: NO_OP noop_request { }
I20250114 02:54:25.754501 1990 raft_consensus.cc:2883] T 266da813b4514329be0ad97d07b7114f P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250114 02:54:25.754729 814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddfbab181b1f474192c664cd0316037f" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:25.755317 814 raft_consensus.cc:2463] T ddfbab181b1f474192c664cd0316037f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 0.
I20250114 02:54:25.758010 814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddfbab181b1f474192c664cd0316037f" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:25.758548 814 raft_consensus.cc:3054] T ddfbab181b1f474192c664cd0316037f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:25.766316 814 raft_consensus.cc:2463] T ddfbab181b1f474192c664cd0316037f P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1.
I20250114 02:54:25.771185 470 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33540 (request call id 242) took 1581 ms. Trace:
I20250114 02:54:25.772814 1207 tablet_bootstrap.cc:492] T e15e6e52d7294aeeb4463b3a76022ec3 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:25.773263 1207 ts_tablet_manager.cc:1397] T e15e6e52d7294aeeb4463b3a76022ec3 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.530s user 0.010s sys 0.000s
I20250114 02:54:25.775418 1207 raft_consensus.cc:357] T e15e6e52d7294aeeb4463b3a76022ec3 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:25.780256 471 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet 1381e26725174b9c877075cf5444f566
I20250114 02:54:25.781973 1206 tablet_bootstrap.cc:492] T 6e858a8449444ef999b9bbef612249d3 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:25.782358 1206 ts_tablet_manager.cc:1397] T 6e858a8449444ef999b9bbef612249d3 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.331s user 0.007s sys 0.015s
I20250114 02:54:25.783555 789 tablet_service.cc:1514] Processing DeleteTablet for tablet 1381e26725174b9c877075cf5444f566 with delete_type TABLET_DATA_DELETED (replaced by tablet 9ffec1d1bb594fda83df8d94d4533d56) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:25.784909 462 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_c778a0b532b64bcdbffa5cca153bcfcc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:25.785025 1206 raft_consensus.cc:357] T 6e858a8449444ef999b9bbef612249d3 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:25.785645 1206 raft_consensus.cc:383] T 6e858a8449444ef999b9bbef612249d3 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:25.785876 1206 raft_consensus.cc:738] T 6e858a8449444ef999b9bbef612249d3 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:25.786662 1206 consensus_queue.cc:260] T 6e858a8449444ef999b9bbef612249d3 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:25.787510 959 raft_consensus.cc:2463] T ddfbab181b1f474192c664cd0316037f P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1.
I20250114 02:54:25.788015 938 tablet_service.cc:1514] Processing DeleteTablet for tablet 1381e26725174b9c877075cf5444f566 with delete_type TABLET_DATA_DELETED (replaced by tablet 9ffec1d1bb594fda83df8d94d4533d56) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:25.791973 471 catalog_manager.cc:6596] ReplaceTablet: tablet 1381e26725174b9c877075cf5444f566 deleted and replaced by tablet 9ffec1d1bb594fda83df8d94d4533d56
I20250114 02:54:25.792423 471 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:33540 (request call id 243) took 1439 ms. Trace:
I20250114 02:54:25.792694 471 rpcz_store.cc:276] 0114 02:54:24.353413 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:25.625117 (+1271704us) service_pool.cc:225] Handling call
0114 02:54:25.792340 (+167223us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:25.648724 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:25.649120 (+ 396us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:25.649154 (+ 34us) write_op.cc:435] Acquired schema lock
0114 02:54:25.649182 (+ 28us) tablet.cc:659] Decoding operations
0114 02:54:25.649507 (+ 325us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:25.649541 (+ 34us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:25.649563 (+ 22us) tablet.cc:682] Acquiring locks for 2 operations
0114 02:54:25.649742 (+ 179us) tablet.cc:698] Row locks acquired
0114 02:54:25.649754 (+ 12us) write_op.cc:260] PREPARE: finished
0114 02:54:25.649849 (+ 95us) write_op.cc:270] Start()
0114 02:54:25.649936 (+ 87us) write_op.cc:276] Timestamp: P: 1736823265649727 usec, L: 0
0114 02:54:25.649958 (+ 22us) op_driver.cc:348] REPLICATION: starting
0114 02:54:25.650394 (+ 436us) log.cc:844] Serialized 659 byte log entry
0114 02:54:25.746423 (+ 96029us) spinlock_profiling.cc:243] Waited 92.2 ms on lock 0x7b58000d01b0. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616cad50 00007f64616bfa94 00007f64616c49d1 00007f64616c4959 00007f64616c48e9 00007f64616c48b1 00007f64616c3a7d 000000000059ea04 000000000059e839 00007f645a7fc866 00007f645a7ffcc1
0114 02:54:25.771313 (+ 24890us) op_driver.cc:464] REPLICATION: finished
0114 02:54:25.771544 (+ 231us) write_op.cc:301] APPLY: starting
0114 02:54:25.771596 (+ 52us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:25.771701 (+ 105us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:25.771720 (+ 19us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:25.772123 (+ 403us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:25.772362 (+ 239us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:25.772418 (+ 56us) write_op.cc:312] APPLY: finished
0114 02:54:25.772585 (+ 167us) log.cc:844] Serialized 29 byte log entry
0114 02:54:25.772734 (+ 149us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:25.772795 (+ 61us) write_op.cc:454] Released schema lock
0114 02:54:25.772826 (+ 31us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":97,"apply.run_cpu_time_us":1474,"apply.run_wall_time_us":1474,"num_ops":2,"prepare.queue_time_us":5332,"prepare.run_cpu_time_us":1976,"prepare.run_wall_time_us":3005,"raft.queue_time_us":85,"raft.run_cpu_time_us":2149,"raft.run_wall_time_us":94671,"replication_time_us":121251,"spinlock_wait_cycles":212097024}]]}
I20250114 02:54:25.798326 471 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_69b77a6fd8ad45cc9e3dc32b202d93cd" } modify_external_catalogs: true
W20250114 02:54:25.798458 898 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.67:45749, user_credentials={real_user=slave}} blocked reactor thread for 64115.7us
W20250114 02:54:25.798894 462 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c778a0b532b64bcdbffa5cca153bcfcc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:25.771517 470 rpcz_store.cc:276] 0114 02:54:24.190265 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:25.498960 (+1308695us) service_pool.cc:225] Handling call
0114 02:54:25.524249 (+ 25289us) spinlock_profiling.cc:243] Waited 14.7 ms on lock 0x7b54000316d4. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616cad50 00007f646174df2a 00007f64633f2dec 00007f64634ec306 00007f6461250044 00007f646124ffd2 00007f646124ff01 00007f646124fe7c 00007f646124f132 00007f645f512dfc 00007f645f512236
0114 02:54:25.572396 (+ 48147us) catalog_manager.cc:2076] Acquired catalog manager lock
0114 02:54:25.613076 (+ 40680us) catalog_manager.cc:2122] Created new table and tablet info
0114 02:54:25.770529 (+157453us) catalog_manager.cc:2182] Wrote table and tablets to system table
0114 02:54:25.770838 (+ 309us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0114 02:54:25.771100 (+ 262us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:25.628917 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:25.629284 (+ 367us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:25.629307 (+ 23us) write_op.cc:435] Acquired schema lock
0114 02:54:25.629319 (+ 12us) tablet.cc:659] Decoding operations
0114 02:54:25.629590 (+ 271us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:25.629625 (+ 35us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:25.629651 (+ 26us) tablet.cc:682] Acquiring locks for 2 operations
0114 02:54:25.629862 (+ 211us) tablet.cc:698] Row locks acquired
0114 02:54:25.629875 (+ 13us) write_op.cc:260] PREPARE: finished
0114 02:54:25.629991 (+ 116us) write_op.cc:270] Start()
0114 02:54:25.630084 (+ 93us) write_op.cc:276] Timestamp: P: 1736823265629865 usec, L: 0
0114 02:54:25.630098 (+ 14us) op_driver.cc:348] REPLICATION: starting
0114 02:54:25.646766 (+ 16668us) spinlock_profiling.cc:243] Waited 13.8 ms on lock 0x7b5400032120. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616ca6d1 00007f64616db020 00007f646174bbd2 00007f64617412d7 00007f6461741114 00007f6461bad3b1 00007f6461bacb88 00007f6461bb1821 00007f6461bb17d9 00007f6461bb1769 00007f6461bb1731
0114 02:54:25.647244 (+ 478us) log.cc:844] Serialized 433 byte log entry
0114 02:54:25.747704 (+100460us) spinlock_profiling.cc:243] Waited 95.4 ms on lock 0x7b58000d01b0. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616cad50 00007f64616bfa94 00007f64616c49d1 00007f64616c4959 00007f64616c48e9 00007f64616c48b1 00007f64616c3a7d 000000000059ea04 000000000059e839 00007f645a7fc866 00007f645a7ffcc1
0114 02:54:25.768409 (+ 20705us) op_driver.cc:464] REPLICATION: finished
0114 02:54:25.768677 (+ 268us) write_op.cc:301] APPLY: starting
0114 02:54:25.768737 (+ 60us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:25.768857 (+ 120us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:25.768886 (+ 29us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:25.769435 (+ 549us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:25.769631 (+ 196us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 02:54:25.769695 (+ 64us) write_op.cc:312] APPLY: finished
0114 02:54:25.769945 (+ 250us) log.cc:844] Serialized 29 byte log entry
0114 02:54:25.770143 (+ 198us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:25.770238 (+ 95us) write_op.cc:454] Released schema lock
0114 02:54:25.770282 (+ 44us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":33735936,"child_traces":[["op",{"apply.queue_time_us":91,"apply.run_cpu_time_us":1843,"apply.run_wall_time_us":1843,"num_ops":2,"prepare.queue_time_us":14419,"prepare.run_cpu_time_us":4257,"prepare.run_wall_time_us":19569,"raft.queue_time_us":96,"raft.run_cpu_time_us":2170,"raft.run_wall_time_us":99145,"replication_time_us":138175,"spinlock_wait_cycles":251244288}]]}
I20250114 02:54:25.800129 1944 consensus_queue.cc:1035] T 044832851f804208b0a371eb0857dbf3 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:25.802523 470 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_e2ae3006a8ca4082aee1af910f435789" } new_table_name: "default.Table_5e2861a7715a47039e64f1a717c57997" modify_external_catalogs: true
W20250114 02:54:25.808300 751 leader_election.cc:343] T 7774c2087422492696a58e0075921ae7 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:25.808806 751 leader_election.cc:304] T 7774c2087422492696a58e0075921ae7 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:25.810802 1206 ts_tablet_manager.cc:1428] T 6e858a8449444ef999b9bbef612249d3 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.028s user 0.006s sys 0.001s
I20250114 02:54:25.812659 789 tablet_service.cc:1467] Processing CreateTablet for tablet 992bb16197b74be5b1dc884f29d9c240 (DEFAULT_TABLE table=default.Table_c62ee80bacab484cb74431dcb4723918 [id=01f1fa2a006c4bd4bc22e6648a2152dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:25.816099 790 tablet_service.cc:1467] Processing CreateTablet for tablet ddad7584337345109ce102702008ab37 (DEFAULT_TABLE table=default.Table_47299fab0314433585117eb56b89c0b6 [id=347f25d3fd324991a401377d1327c93f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:25.816359 797 tablet_service.cc:1467] Processing CreateTablet for tablet dbb1d0851221497ca2b41844526b4502 (DEFAULT_TABLE table=default.Table_a7de17bb754b4696bb8a0b05175061ad [id=efc5c73b0c1543e3ba0dbbf3c93481ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:25.816807 1741 raft_consensus.cc:2743] T 7774c2087422492696a58e0075921ae7 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:25.817862 751 leader_election.cc:304] T 5ac7708550954a209e2a8d217c3b39bd P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters: 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:25.819201 932 tablet_service.cc:1467] Processing CreateTablet for tablet dbb1d0851221497ca2b41844526b4502 (DEFAULT_TABLE table=default.Table_a7de17bb754b4696bb8a0b05175061ad [id=efc5c73b0c1543e3ba0dbbf3c93481ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:25.821017 932 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbb1d0851221497ca2b41844526b4502. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:25.824980 933 tablet_service.cc:1467] Processing CreateTablet for tablet ddad7584337345109ce102702008ab37 (DEFAULT_TABLE table=default.Table_47299fab0314433585117eb56b89c0b6 [id=347f25d3fd324991a401377d1327c93f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:25.826210 933 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddad7584337345109ce102702008ab37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:25.839268 466 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet de4ca83a69704a368a25954cdc1b9392 (table default.Table_69b77a6fd8ad45cc9e3dc32b202d93cd [id=8536eb7f6bf542daa88a8db0e4873456]). Expected version 3 got 2
I20250114 02:54:25.840569 466 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 5ac7708550954a209e2a8d217c3b39bd (table default.Table_ee814dd266e64286864db0eac9e11664 [id=13151866bf2b4f44b1dd5c92380a4d43]). Expected version 1 got 0
I20250114 02:54:25.842875 790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddad7584337345109ce102702008ab37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:25.844645 1741 raft_consensus.cc:2798] T 5ac7708550954a209e2a8d217c3b39bd P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:25.845130 1741 raft_consensus.cc:491] T 5ac7708550954a209e2a8d217c3b39bd P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:25.845475 1741 raft_consensus.cc:3054] T 5ac7708550954a209e2a8d217c3b39bd P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:25.849225 938 tablet_service.cc:1467] Processing CreateTablet for tablet 992bb16197b74be5b1dc884f29d9c240 (DEFAULT_TABLE table=default.Table_c62ee80bacab484cb74431dcb4723918 [id=01f1fa2a006c4bd4bc22e6648a2152dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:25.850309 938 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 992bb16197b74be5b1dc884f29d9c240. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:25.788954 1040 leader_election.cc:304] T ddfbab181b1f474192c664cd0316037f P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:25.853425 1966 raft_consensus.cc:2798] T ddfbab181b1f474192c664cd0316037f P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:25.853869 1966 raft_consensus.cc:695] T ddfbab181b1f474192c664cd0316037f P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Becoming Leader. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Running, Role: LEADER
I20250114 02:54:25.854620 1966 consensus_queue.cc:237] T ddfbab181b1f474192c664cd0316037f P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:25.809787 471 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_69b77a6fd8ad45cc9e3dc32b202d93cd" } modify_external_catalogs: true
I20250114 02:54:25.841063 465 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_96b6a43fc44a43ce9f6485f0e8daae5d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:25.858348 1040 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.66:44037, user_credentials={real_user=slave}} blocked reactor thread for 69470.2us
W20250114 02:54:25.859642 465 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_96b6a43fc44a43ce9f6485f0e8daae5d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:25.857167 468 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_8826c1ea171946ab9bc9b3572bf349c4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:25.861416 468 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8826c1ea171946ab9bc9b3572bf349c4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:25.862121 1083 tablet_service.cc:1467] Processing CreateTablet for tablet 992bb16197b74be5b1dc884f29d9c240 (DEFAULT_TABLE table=default.Table_c62ee80bacab484cb74431dcb4723918 [id=01f1fa2a006c4bd4bc22e6648a2152dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:25.863023 1083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 992bb16197b74be5b1dc884f29d9c240. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:25.865006 1081 tablet_service.cc:1514] Processing DeleteTablet for tablet 1381e26725174b9c877075cf5444f566 with delete_type TABLET_DATA_DELETED (replaced by tablet 9ffec1d1bb594fda83df8d94d4533d56) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:25.866192 1084 tablet_service.cc:1467] Processing CreateTablet for tablet ddad7584337345109ce102702008ab37 (DEFAULT_TABLE table=default.Table_47299fab0314433585117eb56b89c0b6 [id=347f25d3fd324991a401377d1327c93f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:25.867238 1084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddad7584337345109ce102702008ab37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:25.876334 462 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33540 (request call id 246) took 1037 ms. Trace:
I20250114 02:54:25.876652 462 rpcz_store.cc:276] 0114 02:54:24.838538 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:25.714261 (+875723us) service_pool.cc:225] Handling call
0114 02:54:25.784743 (+ 70482us) spinlock_profiling.cc:243] Waited 55.7 ms on lock 0x7b54000316d4. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616cad50 00007f646174df2a 00007f64633f2dec 00007f64634ec306 00007f6461250044 00007f646124ffd2 00007f646124ff01 00007f646124fe7c 00007f646124f132 00007f645f512dfc 00007f645f512236
0114 02:54:25.805929 (+ 21186us) catalog_manager.cc:2076] Acquired catalog manager lock
0114 02:54:25.830238 (+ 24309us) catalog_manager.cc:2122] Created new table and tablet info
0114 02:54:25.850625 (+ 20387us) catalog_manager.cc:2182] Wrote table and tablets to system table
0114 02:54:25.875976 (+ 25351us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0114 02:54:25.876259 (+ 283us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:25.831815 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:25.832195 (+ 380us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:25.832295 (+ 100us) write_op.cc:435] Acquired schema lock
0114 02:54:25.832312 (+ 17us) tablet.cc:659] Decoding operations
0114 02:54:25.832617 (+ 305us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:25.832665 (+ 48us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:25.832692 (+ 27us) tablet.cc:682] Acquiring locks for 2 operations
0114 02:54:25.832915 (+ 223us) tablet.cc:698] Row locks acquired
0114 02:54:25.832932 (+ 17us) write_op.cc:260] PREPARE: finished
0114 02:54:25.833058 (+ 126us) write_op.cc:270] Start()
0114 02:54:25.833171 (+ 113us) write_op.cc:276] Timestamp: P: 1736823265832909 usec, L: 0
0114 02:54:25.833190 (+ 19us) op_driver.cc:348] REPLICATION: starting
0114 02:54:25.833742 (+ 552us) log.cc:844] Serialized 433 byte log entry
0114 02:54:25.847037 (+ 13295us) op_driver.cc:464] REPLICATION: finished
0114 02:54:25.847274 (+ 237us) write_op.cc:301] APPLY: starting
0114 02:54:25.847327 (+ 53us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:25.847444 (+ 117us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:25.847535 (+ 91us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:25.847955 (+ 420us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:25.848124 (+ 169us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 02:54:25.848168 (+ 44us) write_op.cc:312] APPLY: finished
0114 02:54:25.848389 (+ 221us) log.cc:844] Serialized 29 byte log entry
0114 02:54:25.848527 (+ 138us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:25.848602 (+ 75us) write_op.cc:454] Released schema lock
0114 02:54:25.848636 (+ 34us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":128089856,"child_traces":[["op",{"apply.queue_time_us":96,"apply.run_cpu_time_us":1552,"apply.run_wall_time_us":1552,"num_ops":2,"prepare.queue_time_us":107,"prepare.run_cpu_time_us":2205,"prepare.run_wall_time_us":2205,"raft.queue_time_us":199,"raft.run_cpu_time_us":689,"raft.run_wall_time_us":1516,"replication_time_us":13738}]]}
I20250114 02:54:25.884725 797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbb1d0851221497ca2b41844526b4502. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:25.888693 789 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 992bb16197b74be5b1dc884f29d9c240. 1 dirs total, 0 dirs full, 0 dirs failed
W20250114 02:54:25.892532 470 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 68981913851243a7a1875fc99efb3c49 consensus state has no leader
I20250114 02:54:25.898067 1081 tablet_service.cc:1467] Processing CreateTablet for tablet dbb1d0851221497ca2b41844526b4502 (DEFAULT_TABLE table=default.Table_a7de17bb754b4696bb8a0b05175061ad [id=efc5c73b0c1543e3ba0dbbf3c93481ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:25.899188 1081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbb1d0851221497ca2b41844526b4502. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:25.901971 466 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 55b68586ef8b45439a7dd8dcd2efcd79 (table default.Table_66f2263162f64250a0b49073abe4bd4c [id=c73ede1a5f664d8eb204fb9a012f8f7a]). Expected version 1 got 0
I20250114 02:54:25.903360 466 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 6448eea21ab2422c97da2494658ded3f (table default.Table_0931c459ae794d2091fcc0f51d031548 [id=e279c31bd63b40b9ade03a3b928bf153]). Expected version 2 got 0
I20250114 02:54:25.933920 465 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33540 (request call id 245) took 1095 ms. Trace:
I20250114 02:54:25.934264 465 rpcz_store.cc:276] 0114 02:54:24.838268 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:25.638300 (+800032us) service_pool.cc:225] Handling call
0114 02:54:25.836092 (+197792us) spinlock_profiling.cc:243] Waited 168 ms on lock 0x7b54000316d4. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616cad50 00007f646174df2a 00007f64633f2dec 00007f64634ec306 00007f6461250044 00007f646124ffd2 00007f646124ff01 00007f646124fe7c 00007f646124f132 00007f645f512dfc 00007f645f512236
0114 02:54:25.882476 (+ 46384us) catalog_manager.cc:2076] Acquired catalog manager lock
0114 02:54:25.915009 (+ 32533us) catalog_manager.cc:2122] Created new table and tablet info
0114 02:54:25.933293 (+ 18284us) catalog_manager.cc:2182] Wrote table and tablets to system table
0114 02:54:25.933599 (+ 306us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0114 02:54:25.933864 (+ 265us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:25.916449 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:25.916947 (+ 498us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:25.916980 (+ 33us) write_op.cc:435] Acquired schema lock
0114 02:54:25.917000 (+ 20us) tablet.cc:659] Decoding operations
0114 02:54:25.917270 (+ 270us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:25.917308 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:25.917332 (+ 24us) tablet.cc:682] Acquiring locks for 2 operations
0114 02:54:25.917549 (+ 217us) tablet.cc:698] Row locks acquired
0114 02:54:25.917562 (+ 13us) write_op.cc:260] PREPARE: finished
0114 02:54:25.917675 (+ 113us) write_op.cc:270] Start()
0114 02:54:25.917760 (+ 85us) write_op.cc:276] Timestamp: P: 1736823265917622 usec, L: 0
0114 02:54:25.917776 (+ 16us) op_driver.cc:348] REPLICATION: starting
0114 02:54:25.918253 (+ 477us) log.cc:844] Serialized 433 byte log entry
0114 02:54:25.926076 (+ 7823us) op_driver.cc:464] REPLICATION: finished
0114 02:54:25.926383 (+ 307us) write_op.cc:301] APPLY: starting
0114 02:54:25.926436 (+ 53us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:25.926542 (+ 106us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:25.926650 (+ 108us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:25.927192 (+ 542us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:25.927392 (+ 200us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 02:54:25.927464 (+ 72us) write_op.cc:312] APPLY: finished
0114 02:54:25.927755 (+ 291us) log.cc:844] Serialized 29 byte log entry
0114 02:54:25.927915 (+ 160us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:25.927989 (+ 74us) write_op.cc:454] Released schema lock
0114 02:54:25.928048 (+ 59us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:25.928296 (+ 248us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:25.928320 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":386380160,"child_traces":[["op",{"apply.queue_time_us":105,"apply.run_cpu_time_us":1867,"apply.run_wall_time_us":1867,"num_ops":2,"prepare.queue_time_us":186,"prepare.run_cpu_time_us":2102,"prepare.run_wall_time_us":2104,"raft.queue_time_us":173,"raft.run_cpu_time_us":666,"raft.run_wall_time_us":664,"replication_time_us":8146}]]}
I20250114 02:54:25.904084 466 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet ce7d6088fc884a248918d2348adcaccf (table default.Table_428bd60f16824ac2b8bfaad6819f76e1 [id=35c5879da4b344118a377335fc6f83de]). Expected version 1 got 0
I20250114 02:54:25.944370 466 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 0304df55431e42e1a999ed17dc35e6c0 (table default.Table_4c3843233dda407395ab9652d5870bc6 [id=0d173fdb502f482fb86b6d3e5fdc66f1]). Expected version 1 got 0
I20250114 02:54:25.949412 465 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_acb5bf9354be440bbf6034cf1c535dab" } new_table_name: "default.Table_e4cc8164591444a4bf1e683f3a609d36" modify_external_catalogs: true
I20250114 02:54:25.960096 466 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 044832851f804208b0a371eb0857dbf3 (table default.Table_0518d8bed7304c86aff32e05e6f07c0d [id=515a2fe3f1e84e82bccc02ef5cecef1b]). Expected version 1 got 0
I20250114 02:54:25.973148 1944 raft_consensus.cc:491] T 84738a6c78b24c1fb64013048763b18f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:25.973610 1944 raft_consensus.cc:513] T 84738a6c78b24c1fb64013048763b18f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:25.975252 1944 leader_election.cc:290] T 84738a6c78b24c1fb64013048763b18f P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
W20250114 02:54:25.975637 898 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.67:45749, user_credentials={real_user=slave}} blocked reactor thread for 132258us
I20250114 02:54:25.976814 1207 raft_consensus.cc:383] T e15e6e52d7294aeeb4463b3a76022ec3 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:25.977121 1207 raft_consensus.cc:738] T e15e6e52d7294aeeb4463b3a76022ec3 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:25.976825 2006 consensus_queue.cc:1035] T 71bd00f7d8ac41839f36fd73997631ad P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:25.977746 1207 consensus_queue.cc:260] T e15e6e52d7294aeeb4463b3a76022ec3 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:25.981818 1207 ts_tablet_manager.cc:1428] T e15e6e52d7294aeeb4463b3a76022ec3 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.208s user 0.004s sys 0.000s
I20250114 02:54:25.982633 1207 tablet_bootstrap.cc:492] T 5ac7708550954a209e2a8d217c3b39bd P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:25.984318 959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e15e6e52d7294aeeb4463b3a76022ec3" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:25.985993 959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc69b66bea54625a4b15867b75986b7" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:25.987538 959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38fe8829d98547fb907ef78099df3d7f" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:25.989045 959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92cd8eccacf34ab7a66b3bbc69df3697" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:25.995225 1206 tablet_bootstrap.cc:492] T a52d6749ff834b62a8e20a2f45935240 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:26.006350 468 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_945582460ea34007972cb70eb35e608a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:26.007817 468 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_945582460ea34007972cb70eb35e608a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:25.807695 1099 raft_consensus.cc:2463] T 42cee9e8e26147b593ed73fca6ac1183 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:25.828364 1990 raft_consensus.cc:2267] T 266da813b4514329be0ad97d07b7114f P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:26.015170 1990 ts_tablet_manager.cc:1905] T 266da813b4514329be0ad97d07b7114f P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:26.052690 817 raft_consensus.cc:2463] T 42cee9e8e26147b593ed73fca6ac1183 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:26.055833 467 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet 5ac7708550954a209e2a8d217c3b39bd
I20250114 02:54:26.057096 467 catalog_manager.cc:6596] ReplaceTablet: tablet 5ac7708550954a209e2a8d217c3b39bd deleted and replaced by tablet d0cc8b9e3aee4292bc69e8053d7e8cc1
I20250114 02:54:26.057440 467 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:33540 (request call id 247) took 1177 ms. Trace:
I20250114 02:54:26.058473 1205 tablet_bootstrap.cc:700] T 92cd8eccacf34ab7a66b3bbc69df3697 P 525198593c8e47afaee54e74267b87e3: Time spent opening tablet: real 0.268s user 0.004s sys 0.000s
I20250114 02:54:26.059196 1205 tablet_bootstrap.cc:654] T 92cd8eccacf34ab7a66b3bbc69df3697 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:26.057252 797 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ac7708550954a209e2a8d217c3b39bd with delete_type TABLET_DATA_DELETED (replaced by tablet d0cc8b9e3aee4292bc69e8053d7e8cc1) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:26.057714 467 rpcz_store.cc:276] 0114 02:54:24.879746 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:25.741488 (+861742us) service_pool.cc:225] Handling call
0114 02:54:25.776746 (+ 35258us) spinlock_profiling.cc:243] Waited 32.3 ms on lock 0x7b54000316d4. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616cad50 00007f646174df2a 00007f64633f2dec 00007f64634eed31 00007f6461270d47 00007f6461270cd2 00007f6461270c01 00007f6461270b7c 00007f646126fe32 00007f645f512dfc 00007f645f512236
0114 02:54:26.057359 (+280613us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:26.004303 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:26.004669 (+ 366us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:26.004693 (+ 24us) write_op.cc:435] Acquired schema lock
0114 02:54:26.004706 (+ 13us) tablet.cc:659] Decoding operations
0114 02:54:26.005005 (+ 299us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:26.005044 (+ 39us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:26.005065 (+ 21us) tablet.cc:682] Acquiring locks for 2 operations
0114 02:54:26.005289 (+ 224us) tablet.cc:698] Row locks acquired
0114 02:54:26.005302 (+ 13us) write_op.cc:260] PREPARE: finished
0114 02:54:26.005421 (+ 119us) write_op.cc:270] Start()
0114 02:54:26.005501 (+ 80us) write_op.cc:276] Timestamp: P: 1736823266005356 usec, L: 0
0114 02:54:26.005515 (+ 14us) op_driver.cc:348] REPLICATION: starting
0114 02:54:26.005969 (+ 454us) log.cc:844] Serialized 625 byte log entry
0114 02:54:26.052902 (+ 46933us) op_driver.cc:464] REPLICATION: finished
0114 02:54:26.053125 (+ 223us) write_op.cc:301] APPLY: starting
0114 02:54:26.053188 (+ 63us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:26.053291 (+ 103us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:26.053380 (+ 89us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:26.053749 (+ 369us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:26.053967 (+ 218us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:26.054008 (+ 41us) write_op.cc:312] APPLY: finished
0114 02:54:26.054180 (+ 172us) log.cc:844] Serialized 29 byte log entry
0114 02:54:26.054324 (+ 144us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:26.054396 (+ 72us) write_op.cc:454] Released schema lock
0114 02:54:26.054431 (+ 35us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:26.054631 (+ 200us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:26.054655 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":74273152,"child_traces":[["op",{"apply.queue_time_us":66,"apply.run_cpu_time_us":1475,"apply.run_wall_time_us":1474,"num_ops":2,"prepare.queue_time_us":4588,"prepare.run_cpu_time_us":1940,"prepare.run_wall_time_us":1941,"raft.queue_time_us":1123,"raft.run_cpu_time_us":657,"raft.run_wall_time_us":657,"replication_time_us":47297}]]}
I20250114 02:54:26.060894 1741 raft_consensus.cc:513] T 5ac7708550954a209e2a8d217c3b39bd P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
W20250114 02:54:26.061422 751 leader_election.cc:343] T e15e6e52d7294aeeb4463b3a76022ec3 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:26.077411 751 leader_election.cc:304] T e15e6e52d7294aeeb4463b3a76022ec3 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
W20250114 02:54:26.078367 751 leader_election.cc:343] T edc69b66bea54625a4b15867b75986b7 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:26.078696 751 leader_election.cc:304] T edc69b66bea54625a4b15867b75986b7 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:26.078778 1741 leader_election.cc:290] T 5ac7708550954a209e2a8d217c3b39bd P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Requested vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:26.079252 1741 raft_consensus.cc:2743] T e15e6e52d7294aeeb4463b3a76022ec3 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:26.079735 1741 raft_consensus.cc:2743] T edc69b66bea54625a4b15867b75986b7 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250114 02:54:26.079960 751 leader_election.cc:343] T 38fe8829d98547fb907ef78099df3d7f P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:26.080322 751 leader_election.cc:304] T 38fe8829d98547fb907ef78099df3d7f P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:26.081249 1206 tablet_bootstrap.cc:654] T a52d6749ff834b62a8e20a2f45935240 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
W20250114 02:54:26.081161 751 leader_election.cc:343] T 92cd8eccacf34ab7a66b3bbc69df3697 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:26.081492 751 leader_election.cc:304] T 92cd8eccacf34ab7a66b3bbc69df3697 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:26.083182 1961 raft_consensus.cc:2743] T 38fe8829d98547fb907ef78099df3d7f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:26.083709 1961 raft_consensus.cc:2743] T 92cd8eccacf34ab7a66b3bbc69df3697 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:26.100390 941 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ac7708550954a209e2a8d217c3b39bd with delete_type TABLET_DATA_DELETED (replaced by tablet d0cc8b9e3aee4292bc69e8053d7e8cc1) from {username='slave'} at 127.0.0.1:55068
W20250114 02:54:26.102007 458 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 5d545ac27a7f49faace0655713405cea (table default.Table_b4dc2961391c43fcb08fa062d4e45b3c [id=491b34b03ce14bd9be16e04915468928], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 5d545ac27a7f49faace0655713405cea consensus state has no leader
I20250114 02:54:26.103334 1992 ts_tablet_manager.cc:1918] T 321b9ad6d1ac4d1c80c7a01faf2015af P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:26.108963 1992 log.cc:1198] T 321b9ad6d1ac4d1c80c7a01faf2015af P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/321b9ad6d1ac4d1c80c7a01faf2015af
I20250114 02:54:26.108074 1206 tablet_bootstrap.cc:492] T a52d6749ff834b62a8e20a2f45935240 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:26.111070 1206 ts_tablet_manager.cc:1397] T a52d6749ff834b62a8e20a2f45935240 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.267s user 0.010s sys 0.002s
I20250114 02:54:26.111845 1992 ts_tablet_manager.cc:1939] T 321b9ad6d1ac4d1c80c7a01faf2015af P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:26.115976 1206 raft_consensus.cc:357] T a52d6749ff834b62a8e20a2f45935240 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:26.123549 1206 raft_consensus.cc:383] T a52d6749ff834b62a8e20a2f45935240 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:26.124271 1206 raft_consensus.cc:738] T a52d6749ff834b62a8e20a2f45935240 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:26.125335 1206 consensus_queue.cc:260] T a52d6749ff834b62a8e20a2f45935240 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:26.127733 468 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_63b25aa4a2aa4e58a1e1b6835d0b25ce" } new_table_name: "default.Table_990b8d5f1bd146d6b358457f954780c9" modify_external_catalogs: true
I20250114 02:54:26.127725 467 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_bacb81fcfeea434abb622c33872e1fde"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:26.133018 1205 tablet_bootstrap.cc:492] T 92cd8eccacf34ab7a66b3bbc69df3697 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:26.121214 1992 tablet_replica.cc:331] T ca986f81ed6848a89588366f3f400eec P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:26.144731 1992 raft_consensus.cc:2238] T ca986f81ed6848a89588366f3f400eec P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 LEADER]: Raft consensus shutting down.
I20250114 02:54:26.145785 1992 raft_consensus.cc:2267] T ca986f81ed6848a89588366f3f400eec P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:26.147742 1992 ts_tablet_manager.cc:1905] T ca986f81ed6848a89588366f3f400eec P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:26.152344 1206 ts_tablet_manager.cc:1428] T a52d6749ff834b62a8e20a2f45935240 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.040s user 0.005s sys 0.001s
I20250114 02:54:26.154219 1206 tablet_bootstrap.cc:492] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:26.163144 1206 tablet_bootstrap.cc:654] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:26.133460 1205 ts_tablet_manager.cc:1397] T 92cd8eccacf34ab7a66b3bbc69df3697 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.394s user 0.011s sys 0.000s
W20250114 02:54:26.151304 898 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.65:39397, user_credentials={real_user=slave}} blocked reactor thread for 83029.4us
I20250114 02:54:26.121320 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet 321b9ad6d1ac4d1c80c7a01faf2015af (table default.Table_efc3b170c00f4ce09b2601f7f5ebf3b8 [id=dba5ff68f085474caeaf34c387ecac6f]) successfully deleted
W20250114 02:54:26.164815 467 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bacb81fcfeea434abb622c33872e1fde in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250114 02:54:26.165019 458 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.107.65:39397, user_credentials={real_user=slave}} blocked reactor thread for 43725.3us
I20250114 02:54:26.166325 470 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_cd7d049881524116bf642f02443f44ad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:26.166476 2007 consensus_queue.cc:1035] T 71bd00f7d8ac41839f36fd73997631ad P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:26.167392 2015 consensus_queue.cc:1035] T 044832851f804208b0a371eb0857dbf3 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250114 02:54:26.167805 470 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cd7d049881524116bf642f02443f44ad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:26.168674 462 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_dd50844b27ef42dda7cab6f276fac825"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:26.169966 462 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dd50844b27ef42dda7cab6f276fac825 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:26.172298 898 leader_election.cc:304] T ce7d6088fc884a248918d2348adcaccf P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 525198593c8e47afaee54e74267b87e3; no voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:26.173058 2015 raft_consensus.cc:2743] T ce7d6088fc884a248918d2348adcaccf P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250114 02:54:26.176144 898 leader_election.cc:343] T 84738a6c78b24c1fb64013048763b18f P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Not found: Tablet not found: 84738a6c78b24c1fb64013048763b18f
I20250114 02:54:26.176738 898 leader_election.cc:304] T 42cee9e8e26147b593ed73fca6ac1183 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
I20250114 02:54:26.180429 1205 raft_consensus.cc:357] T 92cd8eccacf34ab7a66b3bbc69df3697 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:26.181041 1205 raft_consensus.cc:383] T 92cd8eccacf34ab7a66b3bbc69df3697 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:26.181352 1205 raft_consensus.cc:738] T 92cd8eccacf34ab7a66b3bbc69df3697 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:26.182013 1205 consensus_queue.cc:260] T 92cd8eccacf34ab7a66b3bbc69df3697 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:26.163538 1741 raft_consensus.cc:491] T a52d6749ff834b62a8e20a2f45935240 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:26.184511 1741 raft_consensus.cc:513] T a52d6749ff834b62a8e20a2f45935240 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:26.186606 1741 leader_election.cc:290] T a52d6749ff834b62a8e20a2f45935240 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
W20250114 02:54:26.186775 2013 log.cc:926] Time spent T ca986f81ed6848a89588366f3f400eec P 525198593c8e47afaee54e74267b87e3: Append to log took a long time: real 0.051s user 0.001s sys 0.000s
I20250114 02:54:26.170794 1948 raft_consensus.cc:491] T 6e858a8449444ef999b9bbef612249d3 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:26.187177 1948 raft_consensus.cc:513] T 6e858a8449444ef999b9bbef612249d3 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:26.189177 1948 leader_election.cc:290] T 6e858a8449444ef999b9bbef612249d3 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:26.193092 2015 raft_consensus.cc:2798] T 42cee9e8e26147b593ed73fca6ac1183 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:26.193531 2015 raft_consensus.cc:695] T 42cee9e8e26147b593ed73fca6ac1183 P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Becoming Leader. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Running, Role: LEADER
I20250114 02:54:26.194504 2015 consensus_queue.cc:237] T 42cee9e8e26147b593ed73fca6ac1183 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:26.199115 1948 consensus_queue.cc:1035] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:26.215469 1992 ts_tablet_manager.cc:1918] T ca986f81ed6848a89588366f3f400eec P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:26.225180 1992 log.cc:1198] T ca986f81ed6848a89588366f3f400eec P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/ca986f81ed6848a89588366f3f400eec
I20250114 02:54:26.226611 1992 ts_tablet_manager.cc:1939] T ca986f81ed6848a89588366f3f400eec P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:26.228754 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet ca986f81ed6848a89588366f3f400eec (table default.Table_123095518d164d3db7fe3bd50af9dbd4 [id=4744eda447ca4785a9c065aca94135e9]) successfully deleted
W20250114 02:54:26.231166 458 catalog_manager.cc:4839] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet 66bfa92db9874750b3c340e31a06c412 because the tablet was not found. No further retry: Not found: Tablet not found: 66bfa92db9874750b3c340e31a06c412
W20250114 02:54:26.234037 458 catalog_manager.cc:4839] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet 0f1ea1608213492c9a216757896ff2a4 because the tablet was not found. No further retry: Not found: Tablet not found: 0f1ea1608213492c9a216757896ff2a4
I20250114 02:54:26.235685 1958 tablet.cc:1721] T de4ca83a69704a368a25954cdc1b9392 P 819f9e3412e54d44abb11d8b3ca877c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20250114 02:54:26.235977 458 catalog_manager.cc:4839] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet 9812754c610548d1b43084bbc94fc3e4 because the tablet was not found. No further retry: Not found: Tablet not found: 9812754c610548d1b43084bbc94fc3e4
W20250114 02:54:26.237944 458 catalog_manager.cc:4839] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet 26328f8e32bc49ed9b224f0e08b9f949 because the tablet was not found. No further retry: Not found: Tablet not found: 26328f8e32bc49ed9b224f0e08b9f949
W20250114 02:54:26.239840 458 catalog_manager.cc:4839] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet 266da813b4514329be0ad97d07b7114f because the tablet was not found. No further retry: Not found: Tablet not found: 266da813b4514329be0ad97d07b7114f
I20250114 02:54:26.220584 959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a52d6749ff834b62a8e20a2f45935240" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:26.242630 1992 tablet_replica.cc:331] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:26.221451 958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e858a8449444ef999b9bbef612249d3" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:26.222872 957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ac7708550954a209e2a8d217c3b39bd" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
I20250114 02:54:26.243903 957 raft_consensus.cc:3054] T 5ac7708550954a209e2a8d217c3b39bd P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:26.245272 1992 raft_consensus.cc:2238] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 LEADER]: Raft consensus shutting down.
I20250114 02:54:26.219916 1965 ts_tablet_manager.cc:1918] T 321b9ad6d1ac4d1c80c7a01faf2015af P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:26.247884 1965 log.cc:1198] T 321b9ad6d1ac4d1c80c7a01faf2015af P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/321b9ad6d1ac4d1c80c7a01faf2015af
I20250114 02:54:26.248760 1965 ts_tablet_manager.cc:1939] T 321b9ad6d1ac4d1c80c7a01faf2015af P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
W20250114 02:54:26.246765 751 leader_election.cc:343] T a52d6749ff834b62a8e20a2f45935240 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
W20250114 02:54:26.252288 751 leader_election.cc:343] T 6e858a8449444ef999b9bbef612249d3 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:26.254475 1206 tablet_bootstrap.cc:492] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:26.230667 1991 tablet.cc:1721] T de4ca83a69704a368a25954cdc1b9392 P 525198593c8e47afaee54e74267b87e3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250114 02:54:26.254866 1206 ts_tablet_manager.cc:1397] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.101s user 0.002s sys 0.017s
I20250114 02:54:26.257153 1206 raft_consensus.cc:357] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:26.257823 1206 raft_consensus.cc:383] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:26.258066 1206 raft_consensus.cc:738] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:26.258673 1206 consensus_queue.cc:260] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:26.261520 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet 321b9ad6d1ac4d1c80c7a01faf2015af (table default.Table_efc3b170c00f4ce09b2601f7f5ebf3b8 [id=dba5ff68f085474caeaf34c387ecac6f]) successfully deleted
I20250114 02:54:26.261982 1206 ts_tablet_manager.cc:1428] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20250114 02:54:26.266800 957 raft_consensus.cc:2463] T 5ac7708550954a209e2a8d217c3b39bd P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 1.
W20250114 02:54:26.273765 456 catalog_manager.cc:4839] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet ba215eb8422d4f879dd0bf5996543962 because the tablet was not found. No further retry: Not found: Tablet not found: ba215eb8422d4f879dd0bf5996543962
I20250114 02:54:26.274680 1992 raft_consensus.cc:2267] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:26.275585 751 leader_election.cc:304] T 5ac7708550954a209e2a8d217c3b39bd P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
W20250114 02:54:26.277387 456 catalog_manager.cc:4839] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet 99b8acb1fe594bf3825c7b9121018db6 because the tablet was not found. No further retry: Not found: Tablet not found: 99b8acb1fe594bf3825c7b9121018db6
I20250114 02:54:26.282058 1741 raft_consensus.cc:2798] T 5ac7708550954a209e2a8d217c3b39bd P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:26.282481 1741 raft_consensus.cc:695] T 5ac7708550954a209e2a8d217c3b39bd P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 LEADER]: Becoming Leader. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Running, Role: LEADER
I20250114 02:54:26.283560 1741 consensus_queue.cc:237] T 5ac7708550954a209e2a8d217c3b39bd P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:26.293159 467 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_b8047d6efb6845768a8d5192b794bd2a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:26.294939 467 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b8047d6efb6845768a8d5192b794bd2a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:26.228688 1205 ts_tablet_manager.cc:1428] T 92cd8eccacf34ab7a66b3bbc69df3697 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.050s user 0.006s sys 0.000s
I20250114 02:54:26.308333 1206 tablet_bootstrap.cc:492] T d05089eae2b74f9fb23a226d358a59da P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:26.315310 1205 tablet_bootstrap.cc:492] T e15e6e52d7294aeeb4463b3a76022ec3 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:26.316860 1992 ts_tablet_manager.cc:1905] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:26.328786 1206 tablet_bootstrap.cc:654] T d05089eae2b74f9fb23a226d358a59da P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
W20250114 02:54:26.329901 456 catalog_manager.cc:4839] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet 290ad07ab8be4791a39194909e000843 because the tablet was not found. No further retry: Not found: Tablet not found: 290ad07ab8be4791a39194909e000843
W20250114 02:54:26.330593 456 catalog_manager.cc:4839] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet 9812754c610548d1b43084bbc94fc3e4 because the tablet was not found. No further retry: Not found: Tablet not found: 9812754c610548d1b43084bbc94fc3e4
W20250114 02:54:26.334717 456 catalog_manager.cc:4839] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet 66bfa92db9874750b3c340e31a06c412 because the tablet was not found. No further retry: Not found: Tablet not found: 66bfa92db9874750b3c340e31a06c412
W20250114 02:54:26.336459 456 catalog_manager.cc:4839] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet bc46815b4803418c95ea9c039a4fae88 because the tablet was not found. No further retry: Not found: Tablet not found: bc46815b4803418c95ea9c039a4fae88
I20250114 02:54:26.341324 1965 tablet_replica.cc:331] T ca986f81ed6848a89588366f3f400eec P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:26.343546 1965 raft_consensus.cc:2238] T ca986f81ed6848a89588366f3f400eec P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:26.347013 469 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet edc69b66bea54625a4b15867b75986b7 (table default.Table_0a0bda55a4a9476c98d961822b8a19bd [id=fb2be5c1ca9444a197f3315ea9c59b09]). Expected version 1 got 0
I20250114 02:54:26.345211 1205 tablet_bootstrap.cc:654] T e15e6e52d7294aeeb4463b3a76022ec3 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:26.350075 469 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet ce7d6088fc884a248918d2348adcaccf (table default.Table_428bd60f16824ac2b8bfaad6819f76e1 [id=35c5879da4b344118a377335fc6f83de]). Expected version 1 got 0
I20250114 02:54:26.347172 1965 raft_consensus.cc:2267] T ca986f81ed6848a89588366f3f400eec P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:26.352802 938 tablet_service.cc:1467] Processing CreateTablet for tablet 5535411d3a27445e99a051b501fe8b39 (DEFAULT_TABLE table=default.Table_08599872e48848e39e5bd7bcbfcec783 [id=ece31fdeafd04267b59a20c4251e5c00]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:26.353883 938 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5535411d3a27445e99a051b501fe8b39. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:26.360352 469 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 0304df55431e42e1a999ed17dc35e6c0 (table default.Table_4c3843233dda407395ab9652d5870bc6 [id=0d173fdb502f482fb86b6d3e5fdc66f1]). Expected version 1 got 0
I20250114 02:54:26.360886 469 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 044832851f804208b0a371eb0857dbf3 (table default.Table_0518d8bed7304c86aff32e05e6f07c0d [id=515a2fe3f1e84e82bccc02ef5cecef1b]). Expected version 1 got 0
I20250114 02:54:26.361631 469 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 92cd8eccacf34ab7a66b3bbc69df3697 (table default.Table_63b25aa4a2aa4e58a1e1b6835d0b25ce [id=26627ccdc808451fb9593ad7a6df6991]). Expected version 1 got 0
I20250114 02:54:26.362236 469 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 38fe8829d98547fb907ef78099df3d7f (table default.Table_60450873174a4c80aa7e2122343d82be [id=9299683349354f879847b7c9d93f6072]). Expected version 1 got 0
W20250114 02:54:26.363232 468 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 92cd8eccacf34ab7a66b3bbc69df3697 consensus state has no leader
I20250114 02:54:26.377280 463 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet 0304df55431e42e1a999ed17dc35e6c0 (table default.Table_4c3843233dda407395ab9652d5870bc6 [id=0d173fdb502f482fb86b6d3e5fdc66f1]). Expected version 1 got 0
I20250114 02:54:26.377593 1965 ts_tablet_manager.cc:1905] T ca986f81ed6848a89588366f3f400eec P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:26.382202 932 tablet_service.cc:1467] Processing CreateTablet for tablet 9ffec1d1bb594fda83df8d94d4533d56 (DEFAULT_TABLE table=default.Table_b624816662024c55a5f5ef5830b20b7c [id=d2ea84ea59ad4b03ad16bb3052a441f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:26.386698 932 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ffec1d1bb594fda83df8d94d4533d56. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:26.071952 1207 tablet_bootstrap.cc:654] T 5ac7708550954a209e2a8d217c3b39bd P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:26.408722 468 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_db23db370ae74af5b334c7e6b5e130c2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:26.410379 468 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_db23db370ae74af5b334c7e6b5e130c2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:26.411513 467 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_403c29800daf456d93b3b60002fa6f96"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:26.413702 1205 tablet_bootstrap.cc:492] T e15e6e52d7294aeeb4463b3a76022ec3 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
W20250114 02:54:26.413558 467 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_403c29800daf456d93b3b60002fa6f96 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:26.414094 1205 ts_tablet_manager.cc:1397] T e15e6e52d7294aeeb4463b3a76022ec3 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.099s user 0.009s sys 0.000s
I20250114 02:54:26.429386 1205 raft_consensus.cc:357] T e15e6e52d7294aeeb4463b3a76022ec3 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:26.430085 1205 raft_consensus.cc:383] T e15e6e52d7294aeeb4463b3a76022ec3 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:26.430330 1205 raft_consensus.cc:738] T e15e6e52d7294aeeb4463b3a76022ec3 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:26.381430 789 tablet_service.cc:1467] Processing CreateTablet for tablet 5535411d3a27445e99a051b501fe8b39 (DEFAULT_TABLE table=default.Table_08599872e48848e39e5bd7bcbfcec783 [id=ece31fdeafd04267b59a20c4251e5c00]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:26.433037 789 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5535411d3a27445e99a051b501fe8b39. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:26.434933 470 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_81dd1d2ab8b741db9483ac53260431ef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:26.382421 797 tablet_service.cc:1467] Processing CreateTablet for tablet 9ffec1d1bb594fda83df8d94d4533d56 (DEFAULT_TABLE table=default.Table_b624816662024c55a5f5ef5830b20b7c [id=d2ea84ea59ad4b03ad16bb3052a441f4]), partition=RANGE (key) PARTITION UNBOUNDED
W20250114 02:54:26.436528 470 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_81dd1d2ab8b741db9483ac53260431ef in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:26.436877 797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ffec1d1bb594fda83df8d94d4533d56. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:26.437955 463 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet 044832851f804208b0a371eb0857dbf3 (table default.Table_0518d8bed7304c86aff32e05e6f07c0d [id=515a2fe3f1e84e82bccc02ef5cecef1b]). Expected version 1 got 0
I20250114 02:54:26.440495 1205 consensus_queue.cc:260] T e15e6e52d7294aeeb4463b3a76022ec3 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:26.446619 1205 ts_tablet_manager.cc:1428] T e15e6e52d7294aeeb4463b3a76022ec3 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.032s user 0.007s sys 0.000s
I20250114 02:54:26.459878 1205 tablet_bootstrap.cc:492] T edc69b66bea54625a4b15867b75986b7 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:26.479918 1205 tablet_bootstrap.cc:654] T edc69b66bea54625a4b15867b75986b7 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:26.481500 1741 raft_consensus.cc:491] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:26.481990 1741 raft_consensus.cc:513] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:26.480090 463 catalog_manager.cc:5526] T ce7d6088fc884a248918d2348adcaccf P 90d6c2c58bb54e0eafb365625ac33871 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67). New cstate: current_term: 1 leader_uuid: "90d6c2c58bb54e0eafb365625ac33871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } }
I20250114 02:54:26.483883 1741 leader_election.cc:290] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:26.485770 957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32dd43e8a3544dbaa6e2fbb4b4c1dc28" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
W20250114 02:54:26.494616 751 leader_election.cc:343] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:26.502198 1990 ts_tablet_manager.cc:1918] T 266da813b4514329be0ad97d07b7114f P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:26.503261 1990 log.cc:1198] T 266da813b4514329be0ad97d07b7114f P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/266da813b4514329be0ad97d07b7114f
I20250114 02:54:26.504060 1965 ts_tablet_manager.cc:1918] T ca986f81ed6848a89588366f3f400eec P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:26.506775 1965 log.cc:1198] T ca986f81ed6848a89588366f3f400eec P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/ca986f81ed6848a89588366f3f400eec
I20250114 02:54:26.507985 1965 ts_tablet_manager.cc:1939] T ca986f81ed6848a89588366f3f400eec P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:26.510066 1205 tablet_bootstrap.cc:492] T edc69b66bea54625a4b15867b75986b7 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:26.513842 1205 ts_tablet_manager.cc:1397] T edc69b66bea54625a4b15867b75986b7 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.054s user 0.010s sys 0.000s
I20250114 02:54:26.484566 463 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet ce7d6088fc884a248918d2348adcaccf (table default.Table_428bd60f16824ac2b8bfaad6819f76e1 [id=35c5879da4b344118a377335fc6f83de]). Expected version 1 got 0
I20250114 02:54:26.527034 471 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet de4ca83a69704a368a25954cdc1b9392
I20250114 02:54:26.529011 471 master_service.cc:946] ReplaceTablet: received request to replace tablet 179248cb5acf4740b268e0ec8b2852f9 from {username='slave'} at 127.0.0.1:33540
I20250114 02:54:26.519834 1081 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ac7708550954a209e2a8d217c3b39bd with delete_type TABLET_DATA_DELETED (replaced by tablet d0cc8b9e3aee4292bc69e8053d7e8cc1) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:26.521214 1084 tablet_service.cc:1467] Processing CreateTablet for tablet 5535411d3a27445e99a051b501fe8b39 (DEFAULT_TABLE table=default.Table_08599872e48848e39e5bd7bcbfcec783 [id=ece31fdeafd04267b59a20c4251e5c00]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:26.519969 1205 raft_consensus.cc:357] T edc69b66bea54625a4b15867b75986b7 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:26.522567 1083 tablet_service.cc:1467] Processing CreateTablet for tablet 9ffec1d1bb594fda83df8d94d4533d56 (DEFAULT_TABLE table=default.Table_b624816662024c55a5f5ef5830b20b7c [id=d2ea84ea59ad4b03ad16bb3052a441f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:26.531328 1084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5535411d3a27445e99a051b501fe8b39. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:26.534010 789 tablet_service.cc:1514] Processing DeleteTablet for tablet de4ca83a69704a368a25954cdc1b9392 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:25 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:26.536598 464 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet d3fb28717c6f4844af84bb817fe7dfc4
I20250114 02:54:26.537278 1083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ffec1d1bb594fda83df8d94d4533d56. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:26.537874 464 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:33540 (request call id 244) took 2184 ms. Trace:
I20250114 02:54:26.538693 469 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: default.Table_66f2263162f64250a0b49073abe4bd4c [id=c73ede1a5f664d8eb204fb9a012f8f7a] alter complete (version 1)
I20250114 02:54:26.522500 1099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ac7708550954a209e2a8d217c3b39bd" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:26.524127 1102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a52d6749ff834b62a8e20a2f45935240" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:26.525444 1101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e858a8449444ef999b9bbef612249d3" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:26.543993 789 tablet_service.cc:1514] Processing DeleteTablet for tablet d3fb28717c6f4844af84bb817fe7dfc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:25 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:26.538137 464 rpcz_store.cc:276] 0114 02:54:24.353749 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:25.633759 (+1280010us) service_pool.cc:225] Handling call
0114 02:54:25.657313 (+ 23554us) spinlock_profiling.cc:243] Waited 22.3 ms on lock 0x7b54000316d4. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616cad50 00007f646174df2a 00007f64633f2dec 00007f64634ec616 00007f6461254b44 00007f6461254ad2 00007f6461254a01 00007f646125497c 00007f6461253c32 00007f645f512dfc 00007f645f512236
0114 02:54:25.658378 (+ 1065us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0114 02:54:25.660022 (+ 1644us) catalog_manager.cc:2753] Modifying in-memory table state
0114 02:54:25.660156 (+ 134us) catalog_manager.cc:2762] Locking tablets
0114 02:54:25.699647 (+ 39491us) catalog_manager.cc:2776] Removing table and tablets from system table
0114 02:54:25.804096 (+104449us) catalog_manager.cc:2794] Removing table from by-name map
0114 02:54:25.807004 (+ 2908us) catalog_manager.cc:2810] Committing in-memory state
0114 02:54:26.536509 (+729505us) catalog_manager.cc:2814] Aborting table tasks
0114 02:54:26.537822 (+ 1313us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:25.777831 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:25.778190 (+ 359us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:25.778210 (+ 20us) write_op.cc:435] Acquired schema lock
0114 02:54:25.778223 (+ 13us) tablet.cc:659] Decoding operations
0114 02:54:25.778526 (+ 303us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:25.778561 (+ 35us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:25.778582 (+ 21us) tablet.cc:682] Acquiring locks for 2 operations
0114 02:54:25.778827 (+ 245us) tablet.cc:698] Row locks acquired
0114 02:54:25.778844 (+ 17us) write_op.cc:260] PREPARE: finished
0114 02:54:25.778949 (+ 105us) write_op.cc:270] Start()
0114 02:54:25.779023 (+ 74us) write_op.cc:276] Timestamp: P: 1736823265778813 usec, L: 0
0114 02:54:25.779037 (+ 14us) op_driver.cc:348] REPLICATION: starting
0114 02:54:25.779464 (+ 427us) log.cc:844] Serialized 762 byte log entry
0114 02:54:25.797670 (+ 18206us) op_driver.cc:464] REPLICATION: finished
0114 02:54:25.802875 (+ 5205us) write_op.cc:301] APPLY: starting
0114 02:54:25.802933 (+ 58us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:25.803036 (+ 103us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:25.803118 (+ 82us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:25.803355 (+ 237us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:25.803496 (+ 141us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0114 02:54:25.803536 (+ 40us) write_op.cc:312] APPLY: finished
0114 02:54:25.803700 (+ 164us) log.cc:844] Serialized 29 byte log entry
0114 02:54:25.803830 (+ 130us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:25.803894 (+ 64us) write_op.cc:454] Released schema lock
0114 02:54:25.803924 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:25.804393 (+ 469us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:25.804422 (+ 29us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":51177728,"child_traces":[["op",{"apply.queue_time_us":5048,"apply.run_cpu_time_us":1265,"apply.run_wall_time_us":1423,"num_ops":2,"prepare.queue_time_us":76199,"prepare.run_cpu_time_us":1813,"prepare.run_wall_time_us":1813,"replication_time_us":18541}]]}
I20250114 02:54:26.546504 464 master_service.cc:946] ReplaceTablet: received request to replace tablet 6d741bfd00784d2d830be3ebe50dc7c4 from {username='slave'} at 127.0.0.1:33540
I20250114 02:54:26.540154 938 tablet_service.cc:1514] Processing DeleteTablet for tablet d3fb28717c6f4844af84bb817fe7dfc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:25 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:26.533811 932 tablet_service.cc:1514] Processing DeleteTablet for tablet de4ca83a69704a368a25954cdc1b9392 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:25 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:26.551856 1990 ts_tablet_manager.cc:1939] T 266da813b4514329be0ad97d07b7114f P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
I20250114 02:54:26.554723 1990 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 235) took 1069 ms. Trace:
I20250114 02:54:26.555299 465 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_6071c0da3a0d4ecc9e063e0bab9712de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:26.556712 465 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6071c0da3a0d4ecc9e063e0bab9712de in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:26.530987 1205 raft_consensus.cc:383] T edc69b66bea54625a4b15867b75986b7 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:26.557838 1205 raft_consensus.cc:738] T edc69b66bea54625a4b15867b75986b7 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:26.559957 1205 consensus_queue.cc:260] T edc69b66bea54625a4b15867b75986b7 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:26.566486 1205 ts_tablet_manager.cc:1428] T edc69b66bea54625a4b15867b75986b7 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.049s user 0.006s sys 0.000s
I20250114 02:54:26.580817 932 tablet_service.cc:1514] Processing DeleteTablet for tablet f0bd9bd5a17d418d8082ed0ebd1bf4cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:19 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:26.601547 932 tablet_service.cc:1514] Processing DeleteTablet for tablet 321b9ad6d1ac4d1c80c7a01faf2015af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:23 UTC) from {username='slave'} at 127.0.0.1:55068
W20250114 02:54:26.604965 753 leader_election.cc:343] T 5ac7708550954a209e2a8d217c3b39bd P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
W20250114 02:54:26.606242 753 leader_election.cc:343] T a52d6749ff834b62a8e20a2f45935240 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:26.606564 753 leader_election.cc:304] T a52d6749ff834b62a8e20a2f45935240 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:26.607244 1961 raft_consensus.cc:2743] T a52d6749ff834b62a8e20a2f45935240 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:26.607699 1965 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55068 (request call id 256) took 1107 ms. Trace:
I20250114 02:54:26.608088 1965 rpcz_store.cc:276] 0114 02:54:25.500716 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:25.500822 (+ 106us) service_pool.cc:225] Handling call
0114 02:54:26.341157 (+840335us) ts_tablet_manager.cc:1145] Deleting tablet ca986f81ed6848a89588366f3f400eec
0114 02:54:26.341203 (+ 46us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:26.449337 (+108134us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:26.503829 (+ 54492us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:26.607629 (+103800us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":839778}
I20250114 02:54:26.609514 1965 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55068 (request call id 253) took 1112 ms. Trace:
I20250114 02:54:26.609831 1965 rpcz_store.cc:276] 0114 02:54:25.496610 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:25.500868 (+ 4258us) service_pool.cc:225] Handling call
0114 02:54:26.609242 (+1108374us) ts_tablet_manager.cc:1145] Deleting tablet 26328f8e32bc49ed9b224f0e08b9f949
0114 02:54:26.609287 (+ 45us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:26.609465 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1107582}
I20250114 02:54:26.610889 1965 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55068 (request call id 252) took 1114 ms. Trace:
W20250114 02:54:26.611166 753 leader_election.cc:343] T 6e858a8449444ef999b9bbef612249d3 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:26.611166 1965 rpcz_store.cc:276] 0114 02:54:25.496320 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:25.501330 (+ 5010us) service_pool.cc:225] Handling call
0114 02:54:26.610650 (+1109320us) ts_tablet_manager.cc:1145] Deleting tablet 53a8bb487ad64157ad3baecf5d866eca
0114 02:54:26.610688 (+ 38us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:26.610832 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1108821}
I20250114 02:54:26.611589 753 leader_election.cc:304] T 6e858a8449444ef999b9bbef612249d3 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:26.612303 1961 raft_consensus.cc:2743] T 6e858a8449444ef999b9bbef612249d3 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:26.616005 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet ca986f81ed6848a89588366f3f400eec (table default.Table_123095518d164d3db7fe3bd50af9dbd4 [id=4744eda447ca4785a9c065aca94135e9]) successfully deleted
W20250114 02:54:26.616636 456 catalog_manager.cc:4839] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet 26328f8e32bc49ed9b224f0e08b9f949 because the tablet was not found. No further retry: Not found: Tablet not found: 26328f8e32bc49ed9b224f0e08b9f949
W20250114 02:54:26.617141 456 catalog_manager.cc:4839] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet 53a8bb487ad64157ad3baecf5d866eca because the tablet was not found. No further retry: Not found: Tablet not found: 53a8bb487ad64157ad3baecf5d866eca
I20250114 02:54:26.618333 1205 tablet_bootstrap.cc:492] T 7774c2087422492696a58e0075921ae7 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:26.629233 1205 tablet_bootstrap.cc:654] T 7774c2087422492696a58e0075921ae7 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:26.612188 1965 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55068 (request call id 259) took 1086 ms. Trace:
I20250114 02:54:26.565346 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32dd43e8a3544dbaa6e2fbb4b4c1dc28" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:26.632505 1965 rpcz_store.cc:276] 0114 02:54:25.526156 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:25.526258 (+ 102us) service_pool.cc:225] Handling call
0114 02:54:26.611962 (+1085704us) ts_tablet_manager.cc:1145] Deleting tablet 0f1ea1608213492c9a216757896ff2a4
0114 02:54:26.612003 (+ 41us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:26.612143 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1085029}
I20250114 02:54:26.633636 1206 tablet_bootstrap.cc:492] T d05089eae2b74f9fb23a226d358a59da P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
W20250114 02:54:26.633797 456 catalog_manager.cc:4839] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet 0f1ea1608213492c9a216757896ff2a4 because the tablet was not found. No further retry: Not found: Tablet not found: 0f1ea1608213492c9a216757896ff2a4
I20250114 02:54:26.634022 1206 ts_tablet_manager.cc:1397] T d05089eae2b74f9fb23a226d358a59da P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.327s user 0.008s sys 0.006s
I20250114 02:54:26.633232 1990 rpcz_store.cc:276] 0114 02:54:25.484938 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:25.487445 (+ 2507us) service_pool.cc:225] Handling call
0114 02:54:25.655810 (+168365us) ts_tablet_manager.cc:1145] Deleting tablet 266da813b4514329be0ad97d07b7114f
0114 02:54:25.655845 (+ 35us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:26.033289 (+377444us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:26.501854 (+468565us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:26.554637 (+ 52783us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":155810}
I20250114 02:54:26.634764 1965 tablet_replica.cc:331] T 46360f726fad4534a2daf67581346fc1 P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:26.635322 1990 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 237) took 1143 ms. Trace:
I20250114 02:54:26.635614 1965 raft_consensus.cc:2238] T 46360f726fad4534a2daf67581346fc1 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:26.636152 1965 raft_consensus.cc:2267] T 46360f726fad4534a2daf67581346fc1 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:26.635780 1990 rpcz_store.cc:276] 0114 02:54:25.491481 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:25.491599 (+ 118us) service_pool.cc:225] Handling call
0114 02:54:26.634773 (+1143174us) ts_tablet_manager.cc:1145] Deleting tablet 66bfa92db9874750b3c340e31a06c412
0114 02:54:26.634842 (+ 69us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:26.635251 (+ 409us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1134503}
I20250114 02:54:26.638242 1965 ts_tablet_manager.cc:1905] T 46360f726fad4534a2daf67581346fc1 P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:26.650749 932 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f1ea1608213492c9a216757896ff2a4 with delete_type TABLET_DATA_DELETED (replaced by tablet 550e84ac4d1b4ef58dfdea6e552ee940) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:26.658412 1958 tablet.cc:1721] T 4bb9b429af92446bbe13894d5dfe4c47 P 819f9e3412e54d44abb11d8b3ca877c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250114 02:54:26.662706 466 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.66:50663 (request call id 48) took 1238 ms (client timeout 1000 ms). Trace:
I20250114 02:54:26.662673 1991 tablet.cc:1721] T 4bb9b429af92446bbe13894d5dfe4c47 P 525198593c8e47afaee54e74267b87e3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250114 02:54:26.663028 466 rpcz_store.cc:269] 0114 02:54:25.424471 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:25.455823 (+ 31352us) service_pool.cc:225] Handling call
0114 02:54:25.516603 (+ 60780us) spinlock_profiling.cc:243] Waited 54.3 ms on lock 0x7b54000316d4. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616cad50 00007f646174df2a 00007f64633f2dec 00007f64634eadee 00007f6461248fc4 00007f6461248f52 00007f6461248e81 00007f6461248dfc 00007f64612480b2 00007f645f512dfc 00007f645f512236
0114 02:54:26.662639 (+1146036us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":26,"spinlock_wait_cycles":124863360}
I20250114 02:54:26.663283 1205 tablet_bootstrap.cc:492] T 7774c2087422492696a58e0075921ae7 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:26.664130 1205 ts_tablet_manager.cc:1397] T 7774c2087422492696a58e0075921ae7 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.046s user 0.007s sys 0.002s
I20250114 02:54:26.668313 470 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_1ebcd922bb294485b52278a74830aaab" } new_table_name: "default.Table_0e0710e3f43e4e859ad32662704ee41c" modify_external_catalogs: true
W20250114 02:54:26.673669 455 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.125:46837, user_credentials={real_user=slave}} blocked reactor thread for 43683.4us
I20250114 02:54:26.636155 1206 raft_consensus.cc:357] T d05089eae2b74f9fb23a226d358a59da P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:26.677908 1206 raft_consensus.cc:383] T d05089eae2b74f9fb23a226d358a59da P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:26.678175 1206 raft_consensus.cc:738] T d05089eae2b74f9fb23a226d358a59da P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:26.689990 464 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet 6d741bfd00784d2d830be3ebe50dc7c4
I20250114 02:54:26.691167 464 catalog_manager.cc:6596] ReplaceTablet: tablet 6d741bfd00784d2d830be3ebe50dc7c4 deleted and replaced by tablet 84edeeba97314e1abef8b13aab838e98
I20250114 02:54:26.692961 932 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d741bfd00784d2d830be3ebe50dc7c4 with delete_type TABLET_DATA_DELETED (replaced by tablet 84edeeba97314e1abef8b13aab838e98) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:26.699054 1206 consensus_queue.cc:260] T d05089eae2b74f9fb23a226d358a59da P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:26.726145 2016 raft_consensus.cc:491] T e15e6e52d7294aeeb4463b3a76022ec3 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:26.726610 2016 raft_consensus.cc:513] T e15e6e52d7294aeeb4463b3a76022ec3 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:26.728699 2016 leader_election.cc:290] T e15e6e52d7294aeeb4463b3a76022ec3 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:26.729329 814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e15e6e52d7294aeeb4463b3a76022ec3" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:26.730075 814 raft_consensus.cc:2463] T e15e6e52d7294aeeb4463b3a76022ec3 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:26.731555 898 leader_election.cc:304] T e15e6e52d7294aeeb4463b3a76022ec3 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
I20250114 02:54:26.732285 2016 raft_consensus.cc:2798] T e15e6e52d7294aeeb4463b3a76022ec3 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:26.732686 2016 raft_consensus.cc:491] T e15e6e52d7294aeeb4463b3a76022ec3 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:26.733067 2016 raft_consensus.cc:3054] T e15e6e52d7294aeeb4463b3a76022ec3 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:26.738519 1205 raft_consensus.cc:357] T 7774c2087422492696a58e0075921ae7 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:26.739178 1205 raft_consensus.cc:383] T 7774c2087422492696a58e0075921ae7 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:26.739440 1205 raft_consensus.cc:738] T 7774c2087422492696a58e0075921ae7 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:26.741770 1992 ts_tablet_manager.cc:1918] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:26.742246 1992 log.cc:1198] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/f0bd9bd5a17d418d8082ed0ebd1bf4cc
I20250114 02:54:26.743122 1992 ts_tablet_manager.cc:1939] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:26.745333 466 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_f842a61f536940588cd46bedc206f2b3" } modify_external_catalogs: true
I20250114 02:54:26.745929 466 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_f842a61f536940588cd46bedc206f2b3" } modify_external_catalogs: true
I20250114 02:54:26.749908 2016 raft_consensus.cc:513] T e15e6e52d7294aeeb4463b3a76022ec3 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:26.753788 464 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_ebe5f78fc2dc4adeacda49c55c5b2147" } modify_external_catalogs: true
I20250114 02:54:26.758517 1206 ts_tablet_manager.cc:1428] T d05089eae2b74f9fb23a226d358a59da P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.124s user 0.003s sys 0.003s
I20250114 02:54:26.759788 467 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_95f4830fe0bc41519e53f94f9514583a" } new_table_name: "default.Table_9da872c9a22e4d07947df8db0524e61f" modify_external_catalogs: true
I20250114 02:54:26.766690 817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e15e6e52d7294aeeb4463b3a76022ec3" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:26.767305 817 raft_consensus.cc:3054] T e15e6e52d7294aeeb4463b3a76022ec3 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:26.740042 1205 consensus_queue.cc:260] T 7774c2087422492696a58e0075921ae7 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:26.794132 817 raft_consensus.cc:2463] T e15e6e52d7294aeeb4463b3a76022ec3 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:26.659166 1207 tablet_bootstrap.cc:492] T 5ac7708550954a209e2a8d217c3b39bd P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:26.798729 1207 ts_tablet_manager.cc:1397] T 5ac7708550954a209e2a8d217c3b39bd P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.816s user 0.012s sys 0.009s
I20250114 02:54:26.799420 898 leader_election.cc:304] T e15e6e52d7294aeeb4463b3a76022ec3 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
I20250114 02:54:26.800288 2015 raft_consensus.cc:2798] T e15e6e52d7294aeeb4463b3a76022ec3 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:26.800510 1205 ts_tablet_manager.cc:1428] T 7774c2087422492696a58e0075921ae7 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.136s user 0.007s sys 0.000s
I20250114 02:54:26.800716 2016 leader_election.cc:290] T e15e6e52d7294aeeb4463b3a76022ec3 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Requested vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:26.705192 1101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84738a6c78b24c1fb64013048763b18f" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:26.713487 1081 tablet_service.cc:1514] Processing DeleteTablet for tablet de4ca83a69704a368a25954cdc1b9392 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:25 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:26.714601 1086 tablet_service.cc:1514] Processing DeleteTablet for tablet d3fb28717c6f4844af84bb817fe7dfc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:25 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:26.796290 1206 tablet_bootstrap.cc:492] T 99b940d872ce4781907517c40fe08781 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:26.801039 2015 raft_consensus.cc:695] T e15e6e52d7294aeeb4463b3a76022ec3 P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Becoming Leader. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Running, Role: LEADER
I20250114 02:54:26.822651 2015 consensus_queue.cc:237] T e15e6e52d7294aeeb4463b3a76022ec3 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:26.823503 1086 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d741bfd00784d2d830be3ebe50dc7c4 with delete_type TABLET_DATA_DELETED (replaced by tablet 84edeeba97314e1abef8b13aab838e98) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:26.801474 1205 tablet_bootstrap.cc:492] T 38fe8829d98547fb907ef78099df3d7f P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
W20250114 02:54:26.825492 470 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 30a56fb2d941470590e17f37b90aef26 consensus state has no leader
I20250114 02:54:26.827718 470 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_e18d67a6f19d4259bee6f447deced18c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:26.829623 470 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e18d67a6f19d4259bee6f447deced18c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:26.834055 464 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_ebe5f78fc2dc4adeacda49c55c5b2147" } modify_external_catalogs: true
I20250114 02:54:26.838590 471 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet 179248cb5acf4740b268e0ec8b2852f9
I20250114 02:54:26.839179 1205 tablet_bootstrap.cc:654] T 38fe8829d98547fb907ef78099df3d7f P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:26.841609 789 tablet_service.cc:1514] Processing DeleteTablet for tablet 179248cb5acf4740b268e0ec8b2852f9 with delete_type TABLET_DATA_DELETED (replaced by tablet efcb1bcd47174cde8c8ab7f99b3bf21e) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:26.842170 471 catalog_manager.cc:6596] ReplaceTablet: tablet 179248cb5acf4740b268e0ec8b2852f9 deleted and replaced by tablet efcb1bcd47174cde8c8ab7f99b3bf21e
I20250114 02:54:26.854038 471 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_3cd1350ab3034d69b3f85470c9ab3e34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:26.855530 932 tablet_service.cc:1514] Processing DeleteTablet for tablet 179248cb5acf4740b268e0ec8b2852f9 with delete_type TABLET_DATA_DELETED (replaced by tablet efcb1bcd47174cde8c8ab7f99b3bf21e) from {username='slave'} at 127.0.0.1:55068
W20250114 02:54:26.857102 471 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3cd1350ab3034d69b3f85470c9ab3e34 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:26.857502 1206 tablet_bootstrap.cc:654] T 99b940d872ce4781907517c40fe08781 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
W20250114 02:54:26.850337 2040 log.cc:926] Time spent T 4bb9b429af92446bbe13894d5dfe4c47 P 525198593c8e47afaee54e74267b87e3: Append to log took a long time: real 0.059s user 0.001s sys 0.000s
I20250114 02:54:26.864750 797 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d741bfd00784d2d830be3ebe50dc7c4 with delete_type TABLET_DATA_DELETED (replaced by tablet 84edeeba97314e1abef8b13aab838e98) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:26.867921 2030 raft_consensus.cc:491] T e15e6e52d7294aeeb4463b3a76022ec3 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:26.868561 2030 raft_consensus.cc:513] T e15e6e52d7294aeeb4463b3a76022ec3 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:26.870765 2030 leader_election.cc:290] T e15e6e52d7294aeeb4463b3a76022ec3 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:26.807180 1207 raft_consensus.cc:357] T 5ac7708550954a209e2a8d217c3b39bd P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:26.885462 1206 tablet_bootstrap.cc:492] T 99b940d872ce4781907517c40fe08781 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:26.885900 1206 ts_tablet_manager.cc:1397] T 99b940d872ce4781907517c40fe08781 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.110s user 0.009s sys 0.000s
I20250114 02:54:26.888892 1205 tablet_bootstrap.cc:492] T 38fe8829d98547fb907ef78099df3d7f P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:26.889312 1205 ts_tablet_manager.cc:1397] T 38fe8829d98547fb907ef78099df3d7f P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.088s user 0.013s sys 0.005s
I20250114 02:54:26.891808 1205 raft_consensus.cc:357] T 38fe8829d98547fb907ef78099df3d7f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:26.892625 1205 raft_consensus.cc:383] T 38fe8829d98547fb907ef78099df3d7f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:26.892915 1205 raft_consensus.cc:738] T 38fe8829d98547fb907ef78099df3d7f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:26.882975 1207 raft_consensus.cc:383] T 5ac7708550954a209e2a8d217c3b39bd P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:26.904857 465 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_26355cc977e54afca705ed400dae8207"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:26.905891 1207 raft_consensus.cc:738] T 5ac7708550954a209e2a8d217c3b39bd P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
W20250114 02:54:26.907205 465 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_26355cc977e54afca705ed400dae8207 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:26.888043 1206 raft_consensus.cc:357] T 99b940d872ce4781907517c40fe08781 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:26.908854 1206 raft_consensus.cc:383] T 99b940d872ce4781907517c40fe08781 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:26.909116 1206 raft_consensus.cc:738] T 99b940d872ce4781907517c40fe08781 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:26.883996 1965 ts_tablet_manager.cc:1918] T 46360f726fad4534a2daf67581346fc1 P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:26.913002 1965 log.cc:1198] T 46360f726fad4534a2daf67581346fc1 P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/46360f726fad4534a2daf67581346fc1
I20250114 02:54:26.913898 1965 ts_tablet_manager.cc:1939] T 46360f726fad4534a2daf67581346fc1 P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:26.913519 1206 consensus_queue.cc:260] T 99b940d872ce4781907517c40fe08781 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:26.914160 1992 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 229) took 1293 ms. Trace:
I20250114 02:54:26.907306 1207 consensus_queue.cc:260] T 5ac7708550954a209e2a8d217c3b39bd P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:26.917119 1965 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55068 (request call id 258) took 1391 ms. Trace:
I20250114 02:54:26.925675 1965 rpcz_store.cc:276] 0114 02:54:25.525863 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:25.526284 (+ 421us) service_pool.cc:225] Handling call
0114 02:54:26.634605 (+1108321us) ts_tablet_manager.cc:1145] Deleting tablet 46360f726fad4534a2daf67581346fc1
0114 02:54:26.634659 (+ 54us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:26.718653 (+ 83994us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:26.883754 (+165101us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:26.917056 (+ 33302us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1107187}
I20250114 02:54:26.914445 1205 consensus_queue.cc:260] T 38fe8829d98547fb907ef78099df3d7f P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:26.932036 1205 ts_tablet_manager.cc:1428] T 38fe8829d98547fb907ef78099df3d7f P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.042s user 0.001s sys 0.005s
I20250114 02:54:26.933048 1205 tablet_bootstrap.cc:492] T 99b940d872ce4781907517c40fe08781 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:26.933593 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet 46360f726fad4534a2daf67581346fc1 (table default.Table_3e7cad29b0c0447ab9539b45ee482232 [id=f65443bbd53c47e184a6d2e9e99dd1b0]) successfully deleted
I20250114 02:54:26.936743 1965 tablet_replica.cc:331] T 1381e26725174b9c877075cf5444f566 P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:26.938150 1965 raft_consensus.cc:2238] T 1381e26725174b9c877075cf5444f566 P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Raft consensus shutting down.
I20250114 02:54:26.944283 1965 raft_consensus.cc:2267] T 1381e26725174b9c877075cf5444f566 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:26.947257 1965 ts_tablet_manager.cc:1905] T 1381e26725174b9c877075cf5444f566 P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:26.951824 1205 tablet_bootstrap.cc:654] T 99b940d872ce4781907517c40fe08781 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:26.954550 2015 raft_consensus.cc:491] T 92cd8eccacf34ab7a66b3bbc69df3697 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:26.964650 2015 raft_consensus.cc:513] T 92cd8eccacf34ab7a66b3bbc69df3697 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:26.966475 2015 leader_election.cc:290] T 92cd8eccacf34ab7a66b3bbc69df3697 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:26.981750 1205 tablet_bootstrap.cc:492] T 99b940d872ce4781907517c40fe08781 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:26.984589 1205 ts_tablet_manager.cc:1397] T 99b940d872ce4781907517c40fe08781 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.052s user 0.009s sys 0.001s
I20250114 02:54:26.905570 1990 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 234) took 1404 ms. Trace:
I20250114 02:54:26.989074 1990 rpcz_store.cc:276] 0114 02:54:25.483908 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:25.490107 (+ 6199us) service_pool.cc:225] Handling call
0114 02:54:26.810748 (+1320641us) ts_tablet_manager.cc:1145] Deleting tablet 26328f8e32bc49ed9b224f0e08b9f949
0114 02:54:26.810916 (+ 168us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:26.888571 (+ 77655us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1310107}
I20250114 02:54:26.995311 2031 consensus_queue.cc:1035] T ce7d6088fc884a248918d2348adcaccf P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:54:26.996536 1104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92cd8eccacf34ab7a66b3bbc69df3697" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
W20250114 02:54:27.002146 898 leader_election.cc:343] T 84738a6c78b24c1fb64013048763b18f P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:27.002511 898 leader_election.cc:304] T 84738a6c78b24c1fb64013048763b18f P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 525198593c8e47afaee54e74267b87e3; no voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:27.003516 2018 raft_consensus.cc:2743] T 84738a6c78b24c1fb64013048763b18f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:27.018220 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet 266da813b4514329be0ad97d07b7114f (table default.Table_ae220acb7e0345bdbb80b11398e4d153 [id=bcf9e834733d49a8bc235cd303a75a35]) successfully deleted
I20250114 02:54:27.019083 1103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e15e6e52d7294aeeb4463b3a76022ec3" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:27.019308 1099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e15e6e52d7294aeeb4463b3a76022ec3" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:27.019798 1103 raft_consensus.cc:3054] T e15e6e52d7294aeeb4463b3a76022ec3 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:27.021772 1205 raft_consensus.cc:357] T 99b940d872ce4781907517c40fe08781 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:27.022447 1205 raft_consensus.cc:383] T 99b940d872ce4781907517c40fe08781 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:27.022683 1205 raft_consensus.cc:738] T 99b940d872ce4781907517c40fe08781 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
W20250114 02:54:27.024020 898 leader_election.cc:343] T 92cd8eccacf34ab7a66b3bbc69df3697 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:27.028383 1205 consensus_queue.cc:260] T 99b940d872ce4781907517c40fe08781 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:27.035476 1205 ts_tablet_manager.cc:1428] T 99b940d872ce4781907517c40fe08781 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.050s user 0.004s sys 0.000s
I20250114 02:54:27.037495 2032 consensus_queue.cc:1035] T ddfbab181b1f474192c664cd0316037f P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:54:27.039911 1205 tablet_bootstrap.cc:492] T 719188f639724851a5fc2b37cb60fdbb P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
W20250114 02:54:27.041249 458 catalog_manager.cc:4839] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 66bfa92db9874750b3c340e31a06c412 because the tablet was not found. No further retry: Not found: Tablet not found: 66bfa92db9874750b3c340e31a06c412
I20250114 02:54:27.043128 1103 raft_consensus.cc:2463] T e15e6e52d7294aeeb4463b3a76022ec3 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:27.061175 957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e15e6e52d7294aeeb4463b3a76022ec3" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
W20250114 02:54:27.066920 458 catalog_manager.cc:4839] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 26328f8e32bc49ed9b224f0e08b9f949 because the tablet was not found. No further retry: Not found: Tablet not found: 26328f8e32bc49ed9b224f0e08b9f949
I20250114 02:54:27.082165 1965 ts_tablet_manager.cc:1918] T 1381e26725174b9c877075cf5444f566 P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250114 02:54:27.082561 1965 log.cc:1198] T 1381e26725174b9c877075cf5444f566 P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/1381e26725174b9c877075cf5444f566
I20250114 02:54:27.083284 1965 ts_tablet_manager.cc:1939] T 1381e26725174b9c877075cf5444f566 P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:27.102133 1205 tablet_bootstrap.cc:654] T 719188f639724851a5fc2b37cb60fdbb P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:27.112202 1207 ts_tablet_manager.cc:1428] T 5ac7708550954a209e2a8d217c3b39bd P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.313s user 0.007s sys 0.001s
I20250114 02:54:27.119401 1965 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55068 (request call id 260) took 1335 ms. Trace:
I20250114 02:54:27.124903 1820 consensus_queue.cc:1035] T ce7d6088fc884a248918d2348adcaccf P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:54:27.135638 1965 rpcz_store.cc:276] 0114 02:54:25.783744 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:25.787859 (+ 4115us) service_pool.cc:225] Handling call
0114 02:54:26.936569 (+1148710us) ts_tablet_manager.cc:1145] Deleting tablet 1381e26725174b9c877075cf5444f566
0114 02:54:26.936621 (+ 52us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:27.018143 (+ 81522us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:27.081942 (+ 63799us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:27.119340 (+ 37398us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1141649}
I20250114 02:54:26.939807 1206 ts_tablet_manager.cc:1428] T 99b940d872ce4781907517c40fe08781 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.054s user 0.006s sys 0.000s
I20250114 02:54:27.141669 1206 tablet_bootstrap.cc:492] T 719188f639724851a5fc2b37cb60fdbb P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:26.968007 817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92cd8eccacf34ab7a66b3bbc69df3697" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:27.144539 817 raft_consensus.cc:2463] T 92cd8eccacf34ab7a66b3bbc69df3697 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:26.998294 1741 raft_consensus.cc:491] T d05089eae2b74f9fb23a226d358a59da P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:27.145870 1741 raft_consensus.cc:513] T d05089eae2b74f9fb23a226d358a59da P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:27.147646 1741 leader_election.cc:290] T d05089eae2b74f9fb23a226d358a59da P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:27.006410 814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e15e6e52d7294aeeb4463b3a76022ec3" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:27.148537 814 raft_consensus.cc:2388] T e15e6e52d7294aeeb4463b3a76022ec3 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 90d6c2c58bb54e0eafb365625ac33871 in current term 1: Already voted for candidate 525198593c8e47afaee54e74267b87e3 in this term.
I20250114 02:54:27.078647 1961 consensus_queue.cc:1035] T 5ac7708550954a209e2a8d217c3b39bd P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:27.151435 1083 tablet_service.cc:1514] Processing DeleteTablet for tablet 179248cb5acf4740b268e0ec8b2852f9 with delete_type TABLET_DATA_DELETED (replaced by tablet efcb1bcd47174cde8c8ab7f99b3bf21e) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:27.152276 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet 1381e26725174b9c877075cf5444f566 (table default.Table_b624816662024c55a5f5ef5830b20b7c [id=d2ea84ea59ad4b03ad16bb3052a441f4]) successfully deleted
I20250114 02:54:27.153164 1965 tablet_replica.cc:331] T 5ac7708550954a209e2a8d217c3b39bd P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
W20250114 02:54:27.017537 753 leader_election.cc:343] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:27.154210 753 leader_election.cc:304] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:27.155102 1961 raft_consensus.cc:2743] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:27.155114 1965 raft_consensus.cc:2238] T 5ac7708550954a209e2a8d217c3b39bd P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:27.155707 1965 raft_consensus.cc:2267] T 5ac7708550954a209e2a8d217c3b39bd P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250114 02:54:27.156055 753 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.67:45749, user_credentials={real_user=slave}} blocked reactor thread for 138545us
I20250114 02:54:27.159464 1965 ts_tablet_manager.cc:1905] T 5ac7708550954a209e2a8d217c3b39bd P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:27.160764 898 leader_election.cc:304] T 92cd8eccacf34ab7a66b3bbc69df3697 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters: 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:27.162503 1205 tablet_bootstrap.cc:492] T 719188f639724851a5fc2b37cb60fdbb P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:27.165328 1042 leader_election.cc:304] T e15e6e52d7294aeeb4463b3a76022ec3 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90d6c2c58bb54e0eafb365625ac33871; no voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2
I20250114 02:54:27.166085 1820 raft_consensus.cc:2743] T e15e6e52d7294aeeb4463b3a76022ec3 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:27.168468 1990 tablet_replica.cc:331] T bc46815b4803418c95ea9c039a4fae88 P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:27.171793 1206 tablet_bootstrap.cc:654] T 719188f639724851a5fc2b37cb60fdbb P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:27.172806 1207 tablet_bootstrap.cc:492] T 92cd8eccacf34ab7a66b3bbc69df3697 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
W20250114 02:54:27.174779 455 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.124:39105, user_credentials={real_user=slave}} blocked reactor thread for 251132us
W20250114 02:54:27.177361 455 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 92cd8eccacf34ab7a66b3bbc69df3697 (table default.Table_990b8d5f1bd146d6b358457f954780c9 [id=26627ccdc808451fb9593ad7a6df6991], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 92cd8eccacf34ab7a66b3bbc69df3697 consensus state has no leader
I20250114 02:54:27.140774 1992 rpcz_store.cc:276] 0114 02:54:25.620574 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:25.620698 (+ 124us) service_pool.cc:225] Handling call
0114 02:54:26.242444 (+621746us) ts_tablet_manager.cc:1145] Deleting tablet f0bd9bd5a17d418d8082ed0ebd1bf4cc
0114 02:54:26.242492 (+ 48us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:26.458404 (+215912us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:26.741540 (+283136us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:26.914098 (+172558us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":621161}
I20250114 02:54:27.179661 1992 tablet_replica.cc:331] T 46360f726fad4534a2daf67581346fc1 P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:27.179857 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet f0bd9bd5a17d418d8082ed0ebd1bf4cc (table default.Table_dc087fb027f74c4a8bd5dede1142d97d [id=971cb3ca793e4945ad07e6be966e01f3]) successfully deleted
I20250114 02:54:27.180876 1992 raft_consensus.cc:2238] T 46360f726fad4534a2daf67581346fc1 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:27.181473 1992 raft_consensus.cc:2267] T 46360f726fad4534a2daf67581346fc1 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:27.181896 464 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet a2c3f3a89b3d47f1b0dfacfac8218805
I20250114 02:54:27.184770 1992 ts_tablet_manager.cc:1905] T 46360f726fad4534a2daf67581346fc1 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:27.185611 797 tablet_service.cc:1514] Processing DeleteTablet for tablet a2c3f3a89b3d47f1b0dfacfac8218805 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:26 UTC) from {username='slave'} at 127.0.0.1:53078
W20250114 02:54:27.189028 751 consensus_peers.cc:487] T 5ac7708550954a209e2a8d217c3b39bd P 819f9e3412e54d44abb11d8b3ca877c2 -> Peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Couldn't send request to peer 525198593c8e47afaee54e74267b87e3. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250114 02:54:27.192755 1990 raft_consensus.cc:2238] T bc46815b4803418c95ea9c039a4fae88 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Raft consensus shutting down.
I20250114 02:54:27.193641 1990 raft_consensus.cc:2267] T bc46815b4803418c95ea9c039a4fae88 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:27.195730 1990 ts_tablet_manager.cc:1905] T bc46815b4803418c95ea9c039a4fae88 P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:27.162355 958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d05089eae2b74f9fb23a226d358a59da" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:27.166416 2058 raft_consensus.cc:2798] T 92cd8eccacf34ab7a66b3bbc69df3697 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:27.183445 932 tablet_service.cc:1514] Processing DeleteTablet for tablet a2c3f3a89b3d47f1b0dfacfac8218805 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:26 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:27.210311 2058 raft_consensus.cc:491] T 92cd8eccacf34ab7a66b3bbc69df3697 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250114 02:54:27.213225 751 leader_election.cc:343] T d05089eae2b74f9fb23a226d358a59da P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:27.215250 470 master_service.cc:946] ReplaceTablet: received request to replace tablet 0b8361cd538046e0bcb4aac821748f67 from {username='slave'} at 127.0.0.1:33540
I20250114 02:54:27.212276 2058 raft_consensus.cc:3054] T 92cd8eccacf34ab7a66b3bbc69df3697 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:27.216094 466 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet ed7f575653c848d7b076c17c38c4a244
I20250114 02:54:27.217613 932 tablet_service.cc:1514] Processing DeleteTablet for tablet ed7f575653c848d7b076c17c38c4a244 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:26 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:27.216969 1206 tablet_bootstrap.cc:492] T 719188f639724851a5fc2b37cb60fdbb P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:27.218554 1206 ts_tablet_manager.cc:1397] T 719188f639724851a5fc2b37cb60fdbb P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.077s user 0.010s sys 0.004s
I20250114 02:54:27.219403 797 tablet_service.cc:1514] Processing DeleteTablet for tablet ed7f575653c848d7b076c17c38c4a244 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:26 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:27.209862 1205 ts_tablet_manager.cc:1397] T 719188f639724851a5fc2b37cb60fdbb P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.170s user 0.017s sys 0.000s
I20250114 02:54:27.225315 465 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_8d2e7b6869da423a949c59081ea3618d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:27.226235 1207 tablet_bootstrap.cc:654] T 92cd8eccacf34ab7a66b3bbc69df3697 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
W20250114 02:54:27.226656 465 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8d2e7b6869da423a949c59081ea3618d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:27.231305 466 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_47299fab0314433585117eb56b89c0b6" } new_table_name: "default.Table_bf613453528d4aad91e5ecf042f8aa21" modify_external_catalogs: true
I20250114 02:54:27.225361 1206 raft_consensus.cc:357] T 719188f639724851a5fc2b37cb60fdbb P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:27.232892 1206 raft_consensus.cc:383] T 719188f639724851a5fc2b37cb60fdbb P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:27.233150 1206 raft_consensus.cc:738] T 719188f639724851a5fc2b37cb60fdbb P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:27.232925 2058 raft_consensus.cc:513] T 92cd8eccacf34ab7a66b3bbc69df3697 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:27.236639 1205 raft_consensus.cc:357] T 719188f639724851a5fc2b37cb60fdbb P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:27.238200 1205 raft_consensus.cc:383] T 719188f639724851a5fc2b37cb60fdbb P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:27.238466 1205 raft_consensus.cc:738] T 719188f639724851a5fc2b37cb60fdbb P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:27.240895 1206 consensus_queue.cc:260] T 719188f639724851a5fc2b37cb60fdbb P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:27.245239 1205 consensus_queue.cc:260] T 719188f639724851a5fc2b37cb60fdbb P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:27.248426 1206 ts_tablet_manager.cc:1428] T 719188f639724851a5fc2b37cb60fdbb P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.030s user 0.005s sys 0.001s
I20250114 02:54:27.258895 1205 ts_tablet_manager.cc:1428] T 719188f639724851a5fc2b37cb60fdbb P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.038s user 0.005s sys 0.000s
W20250114 02:54:27.264294 455 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.125:46837, user_credentials={real_user=slave}} blocked reactor thread for 59871.3us
I20250114 02:54:27.264945 1206 tablet_bootstrap.cc:492] T 6e2fb05f72b743e2a02ed36b63f80b23 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:27.272747 1990 ts_tablet_manager.cc:1918] T bc46815b4803418c95ea9c039a4fae88 P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:27.273133 1990 log.cc:1198] T bc46815b4803418c95ea9c039a4fae88 P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/bc46815b4803418c95ea9c039a4fae88
I20250114 02:54:27.273855 1990 ts_tablet_manager.cc:1939] T bc46815b4803418c95ea9c039a4fae88 P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
I20250114 02:54:27.275923 1990 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 244) took 1746 ms. Trace:
I20250114 02:54:27.278481 2016 consensus_queue.cc:1035] T 42cee9e8e26147b593ed73fca6ac1183 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250114 02:54:27.278955 1042 consensus_peers.cc:487] T 46360f726fad4534a2daf67581346fc1 P 90d6c2c58bb54e0eafb365625ac33871 -> Peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Couldn't send request to peer 819f9e3412e54d44abb11d8b3ca877c2. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250114 02:54:27.281996 1207 tablet_bootstrap.cc:492] T 92cd8eccacf34ab7a66b3bbc69df3697 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:27.282452 1207 ts_tablet_manager.cc:1397] T 92cd8eccacf34ab7a66b3bbc69df3697 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.110s user 0.009s sys 0.001s
I20250114 02:54:27.285135 2066 tablet.cc:1721] T de4ca83a69704a368a25954cdc1b9392 P 90d6c2c58bb54e0eafb365625ac33871: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250114 02:54:27.285323 1207 raft_consensus.cc:357] T 92cd8eccacf34ab7a66b3bbc69df3697 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:27.290076 2058 leader_election.cc:290] T 92cd8eccacf34ab7a66b3bbc69df3697 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Requested vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:27.290133 817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92cd8eccacf34ab7a66b3bbc69df3697" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:27.290683 817 raft_consensus.cc:3054] T 92cd8eccacf34ab7a66b3bbc69df3697 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:27.296409 1207 raft_consensus.cc:383] T 92cd8eccacf34ab7a66b3bbc69df3697 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:27.296691 1207 raft_consensus.cc:738] T 92cd8eccacf34ab7a66b3bbc69df3697 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:27.297283 1207 consensus_queue.cc:260] T 92cd8eccacf34ab7a66b3bbc69df3697 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:27.297688 817 raft_consensus.cc:2463] T 92cd8eccacf34ab7a66b3bbc69df3697 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:27.299425 898 leader_election.cc:304] T 92cd8eccacf34ab7a66b3bbc69df3697 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
I20250114 02:54:27.301002 2015 raft_consensus.cc:2798] T 92cd8eccacf34ab7a66b3bbc69df3697 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:27.301390 2015 raft_consensus.cc:695] T 92cd8eccacf34ab7a66b3bbc69df3697 P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Becoming Leader. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Running, Role: LEADER
I20250114 02:54:27.302178 2015 consensus_queue.cc:237] T 92cd8eccacf34ab7a66b3bbc69df3697 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:27.303885 1206 tablet_bootstrap.cc:654] T 6e2fb05f72b743e2a02ed36b63f80b23 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:27.315475 1990 rpcz_store.cc:276] 0114 02:54:25.529215 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:25.529316 (+ 101us) service_pool.cc:225] Handling call
0114 02:54:26.989980 (+1460664us) ts_tablet_manager.cc:1145] Deleting tablet bc46815b4803418c95ea9c039a4fae88
0114 02:54:27.168295 (+178315us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:27.208707 (+ 40412us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:27.272529 (+ 63822us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:27.275867 (+ 3338us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23208,"tablet-delete.queue_time_us":1460078}
I20250114 02:54:27.316870 1990 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 246) took 1787 ms. Trace:
I20250114 02:54:27.317186 1990 rpcz_store.cc:276] 0114 02:54:25.529782 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:25.529971 (+ 189us) service_pool.cc:225] Handling call
0114 02:54:27.316583 (+1786612us) ts_tablet_manager.cc:1145] Deleting tablet bc46815b4803418c95ea9c039a4fae88
0114 02:54:27.316634 (+ 51us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:27.316820 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1786065}
I20250114 02:54:27.318154 1990 tablet_replica.cc:331] T ca986f81ed6848a89588366f3f400eec P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:27.318928 1990 raft_consensus.cc:2238] T ca986f81ed6848a89588366f3f400eec P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:27.328745 1965 ts_tablet_manager.cc:1918] T 5ac7708550954a209e2a8d217c3b39bd P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:27.330401 1965 log.cc:1198] T 5ac7708550954a209e2a8d217c3b39bd P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/5ac7708550954a209e2a8d217c3b39bd
I20250114 02:54:27.331521 1741 raft_consensus.cc:491] T 99b940d872ce4781907517c40fe08781 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:27.332259 1206 tablet_bootstrap.cc:492] T 6e2fb05f72b743e2a02ed36b63f80b23 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:27.331987 1741 raft_consensus.cc:513] T 99b940d872ce4781907517c40fe08781 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:27.333307 1965 ts_tablet_manager.cc:1939] T 5ac7708550954a209e2a8d217c3b39bd P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:27.334527 1741 leader_election.cc:290] T 99b940d872ce4781907517c40fe08781 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:27.334776 1206 ts_tablet_manager.cc:1397] T 6e2fb05f72b743e2a02ed36b63f80b23 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.070s user 0.009s sys 0.000s
I20250114 02:54:27.336093 2015 consensus_queue.cc:1035] T 42cee9e8e26147b593ed73fca6ac1183 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:54:27.339432 471 master_service.cc:946] ReplaceTablet: received request to replace tablet ddad7584337345109ce102702008ab37 from {username='slave'} at 127.0.0.1:33540
I20250114 02:54:27.340258 1102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92cd8eccacf34ab7a66b3bbc69df3697" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:27.340858 1102 raft_consensus.cc:3054] T 92cd8eccacf34ab7a66b3bbc69df3697 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:27.341996 1206 raft_consensus.cc:357] T 6e2fb05f72b743e2a02ed36b63f80b23 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:27.344514 1205 tablet_bootstrap.cc:492] T c86562688d5b4fffb71cbdf7be0afc37 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:27.345008 1206 raft_consensus.cc:383] T 6e2fb05f72b743e2a02ed36b63f80b23 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:27.345259 1206 raft_consensus.cc:738] T 6e2fb05f72b743e2a02ed36b63f80b23 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:27.345784 1206 consensus_queue.cc:260] T 6e2fb05f72b743e2a02ed36b63f80b23 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:27.350354 1205 tablet_bootstrap.cc:654] T c86562688d5b4fffb71cbdf7be0afc37 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:27.354754 957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99b940d872ce4781907517c40fe08781" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:27.355351 957 raft_consensus.cc:2463] T 99b940d872ce4781907517c40fe08781 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 0.
I20250114 02:54:27.355615 1206 ts_tablet_manager.cc:1428] T 6e2fb05f72b743e2a02ed36b63f80b23 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.021s user 0.005s sys 0.000s
I20250114 02:54:27.357896 1965 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55068 (request call id 264) took 1258 ms. Trace:
I20250114 02:54:27.358305 1965 rpcz_store.cc:276] 0114 02:54:26.099503 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:26.099625 (+ 122us) service_pool.cc:225] Handling call
0114 02:54:27.152997 (+1053372us) ts_tablet_manager.cc:1145] Deleting tablet 5ac7708550954a209e2a8d217c3b39bd
0114 02:54:27.153041 (+ 44us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:27.260718 (+107677us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:27.328522 (+ 67804us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:27.357839 (+ 29317us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1052170}
I20250114 02:54:27.361225 1820 consensus_queue.cc:1035] T ddfbab181b1f474192c664cd0316037f P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:27.363322 1990 raft_consensus.cc:2267] T ca986f81ed6848a89588366f3f400eec P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:27.364547 1207 ts_tablet_manager.cc:1428] T 92cd8eccacf34ab7a66b3bbc69df3697 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.082s user 0.006s sys 0.000s
I20250114 02:54:27.365454 1207 tablet_bootstrap.cc:492] T 84738a6c78b24c1fb64013048763b18f P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:27.365657 1990 ts_tablet_manager.cc:1905] T ca986f81ed6848a89588366f3f400eec P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:27.372759 1206 tablet_bootstrap.cc:492] T c86562688d5b4fffb71cbdf7be0afc37 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:27.380113 1207 tablet_bootstrap.cc:654] T 84738a6c78b24c1fb64013048763b18f P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
W20250114 02:54:27.387136 1040 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.66:44037, user_credentials={real_user=slave}} blocked reactor thread for 98458.2us
I20250114 02:54:27.390183 1083 tablet_service.cc:1514] Processing DeleteTablet for tablet a2c3f3a89b3d47f1b0dfacfac8218805 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:26 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:27.396709 751 leader_election.cc:304] T 99b940d872ce4781907517c40fe08781 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
I20250114 02:54:27.397529 1741 raft_consensus.cc:2798] T 99b940d872ce4781907517c40fe08781 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:27.397840 1741 raft_consensus.cc:491] T 99b940d872ce4781907517c40fe08781 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:27.398108 1741 raft_consensus.cc:3054] T 99b940d872ce4781907517c40fe08781 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:27.392127 1206 tablet_bootstrap.cc:654] T c86562688d5b4fffb71cbdf7be0afc37 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:27.405054 1741 raft_consensus.cc:513] T 99b940d872ce4781907517c40fe08781 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:27.409822 1741 leader_election.cc:290] T 99b940d872ce4781907517c40fe08781 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Requested vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:27.412644 1206 tablet_bootstrap.cc:492] T c86562688d5b4fffb71cbdf7be0afc37 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:27.413023 1206 ts_tablet_manager.cc:1397] T c86562688d5b4fffb71cbdf7be0afc37 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.040s user 0.009s sys 0.000s
I20250114 02:54:27.423592 1990 ts_tablet_manager.cc:1918] T ca986f81ed6848a89588366f3f400eec P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:27.423971 1990 log.cc:1198] T ca986f81ed6848a89588366f3f400eec P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/ca986f81ed6848a89588366f3f400eec
I20250114 02:54:27.438498 1083 tablet_service.cc:1514] Processing DeleteTablet for tablet ed7f575653c848d7b076c17c38c4a244 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:26 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:27.439297 1207 tablet_bootstrap.cc:492] T 84738a6c78b24c1fb64013048763b18f P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:27.439664 1207 ts_tablet_manager.cc:1397] T 84738a6c78b24c1fb64013048763b18f P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.074s user 0.003s sys 0.007s
I20250114 02:54:27.443642 1207 raft_consensus.cc:357] T 84738a6c78b24c1fb64013048763b18f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:27.445865 1206 raft_consensus.cc:357] T c86562688d5b4fffb71cbdf7be0afc37 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:27.446481 1206 raft_consensus.cc:383] T c86562688d5b4fffb71cbdf7be0afc37 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:27.446722 1206 raft_consensus.cc:738] T c86562688d5b4fffb71cbdf7be0afc37 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:27.447772 1104 raft_consensus.cc:3054] T 5ac7708550954a209e2a8d217c3b39bd P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:27.449468 1990 ts_tablet_manager.cc:1939] T ca986f81ed6848a89588366f3f400eec P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
W20250114 02:54:27.450677 466 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ddad7584337345109ce102702008ab37 consensus state has no leader
I20250114 02:54:27.444288 1207 raft_consensus.cc:383] T 84738a6c78b24c1fb64013048763b18f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:27.451684 466 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_b624816662024c55a5f5ef5830b20b7c" } modify_external_catalogs: true
I20250114 02:54:27.451933 1207 raft_consensus.cc:738] T 84738a6c78b24c1fb64013048763b18f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:27.452175 466 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_b624816662024c55a5f5ef5830b20b7c" } modify_external_catalogs: true
I20250114 02:54:27.454495 465 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_42771479a01b45f7ade52d2ec3896c4a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:27.455904 465 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_42771479a01b45f7ade52d2ec3896c4a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:27.457530 1207 consensus_queue.cc:260] T 84738a6c78b24c1fb64013048763b18f P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:27.457855 1101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d05089eae2b74f9fb23a226d358a59da" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:27.456005 1206 consensus_queue.cc:260] T c86562688d5b4fffb71cbdf7be0afc37 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:27.414100 2016 raft_consensus.cc:491] T edc69b66bea54625a4b15867b75986b7 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:27.459791 2016 raft_consensus.cc:513] T edc69b66bea54625a4b15867b75986b7 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:27.461567 2016 leader_election.cc:290] T edc69b66bea54625a4b15867b75986b7 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:27.418236 959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99b940d872ce4781907517c40fe08781" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
I20250114 02:54:27.462211 959 raft_consensus.cc:3054] T 99b940d872ce4781907517c40fe08781 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:27.470160 959 raft_consensus.cc:2463] T 99b940d872ce4781907517c40fe08781 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 1.
I20250114 02:54:27.471174 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet 5ac7708550954a209e2a8d217c3b39bd (table default.Table_ee814dd266e64286864db0eac9e11664 [id=13151866bf2b4f44b1dd5c92380a4d43]) successfully deleted
I20250114 02:54:27.471681 1207 ts_tablet_manager.cc:1428] T 84738a6c78b24c1fb64013048763b18f P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.032s user 0.004s sys 0.002s
I20250114 02:54:27.472996 1206 ts_tablet_manager.cc:1428] T c86562688d5b4fffb71cbdf7be0afc37 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.054s user 0.007s sys 0.000s
I20250114 02:54:27.473843 1206 tablet_bootstrap.cc:492] T 59a46bc865cc48df8e98598764fd1b54 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:27.475728 2084 tablet.cc:1721] T 4bb9b429af92446bbe13894d5dfe4c47 P 90d6c2c58bb54e0eafb365625ac33871: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250114 02:54:27.477180 1207 tablet_bootstrap.cc:492] T 7774c2087422492696a58e0075921ae7 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:27.488328 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc69b66bea54625a4b15867b75986b7" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:27.488924 818 raft_consensus.cc:2463] T edc69b66bea54625a4b15867b75986b7 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:27.491844 1101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc69b66bea54625a4b15867b75986b7" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:27.437357 1205 tablet_bootstrap.cc:492] T c86562688d5b4fffb71cbdf7be0afc37 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:27.494958 1205 ts_tablet_manager.cc:1397] T c86562688d5b4fffb71cbdf7be0afc37 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.151s user 0.002s sys 0.010s
I20250114 02:54:27.497673 1205 raft_consensus.cc:357] T c86562688d5b4fffb71cbdf7be0afc37 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:27.498852 1207 tablet_bootstrap.cc:654] T 7774c2087422492696a58e0075921ae7 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:27.501209 2031 raft_consensus.cc:491] T 5ac7708550954a209e2a8d217c3b39bd P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:27.501781 1206 tablet_bootstrap.cc:654] T 59a46bc865cc48df8e98598764fd1b54 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:27.501680 2031 raft_consensus.cc:513] T 5ac7708550954a209e2a8d217c3b39bd P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:27.504442 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ac7708550954a209e2a8d217c3b39bd" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:27.506750 2031 leader_election.cc:290] T 5ac7708550954a209e2a8d217c3b39bd P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:27.513032 1102 raft_consensus.cc:2463] T 92cd8eccacf34ab7a66b3bbc69df3697 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:27.514582 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet bc46815b4803418c95ea9c039a4fae88 (table default.Table_6a4fe39298514187b65def5841d3a992 [id=5c5a7974ad2f4e58ba5c47a6eeaaac62]) successfully deleted
W20250114 02:54:27.515714 458 catalog_manager.cc:4839] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet bc46815b4803418c95ea9c039a4fae88 because the tablet was not found. No further retry: Not found: Tablet not found: bc46815b4803418c95ea9c039a4fae88
I20250114 02:54:27.522794 797 tablet_service.cc:1467] Processing CreateTablet for tablet 53983c5af2444e419ccdd0be395bffa9 (DEFAULT_TABLE table=default.Table_b8047d6efb6845768a8d5192b794bd2a [id=24be7551945c40bc84b668b8747cf0e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:27.523701 797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53983c5af2444e419ccdd0be395bffa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:27.525033 751 leader_election.cc:304] T 99b940d872ce4781907517c40fe08781 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
I20250114 02:54:27.525758 1741 raft_consensus.cc:2798] T 99b940d872ce4781907517c40fe08781 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:27.526157 1741 raft_consensus.cc:695] T 99b940d872ce4781907517c40fe08781 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 LEADER]: Becoming Leader. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Running, Role: LEADER
I20250114 02:54:27.526908 1741 consensus_queue.cc:237] T 99b940d872ce4781907517c40fe08781 P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:27.529174 789 tablet_service.cc:1467] Processing CreateTablet for tablet 587a5f39282a40d0a8c1591fdf36f341 (DEFAULT_TABLE table=default.Table_bacb81fcfeea434abb622c33872e1fde [id=8d818f58d351405f999498f48150fb48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:27.530117 789 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 587a5f39282a40d0a8c1591fdf36f341. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:27.532999 790 tablet_service.cc:1467] Processing CreateTablet for tablet e285f86c29654b9697427516e99ca43a (DEFAULT_TABLE table=default.Table_dd50844b27ef42dda7cab6f276fac825 [id=af4dd8face234ca99bf7523004217964]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:27.534209 790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e285f86c29654b9697427516e99ca43a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:27.578279 1961 consensus_queue.cc:1035] T 5ac7708550954a209e2a8d217c3b39bd P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:54:27.593664 470 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet 0b8361cd538046e0bcb4aac821748f67
W20250114 02:54:27.594614 753 leader_election.cc:343] T d05089eae2b74f9fb23a226d358a59da P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:27.594966 470 catalog_manager.cc:6596] ReplaceTablet: tablet 0b8361cd538046e0bcb4aac821748f67 deleted and replaced by tablet f0e6054998df447287fdff45a6504050
I20250114 02:54:27.595062 753 leader_election.cc:304] T d05089eae2b74f9fb23a226d358a59da P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:27.596726 1961 raft_consensus.cc:2743] T d05089eae2b74f9fb23a226d358a59da P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:27.601002 796 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b8361cd538046e0bcb4aac821748f67 with delete_type TABLET_DATA_DELETED (replaced by tablet f0e6054998df447287fdff45a6504050) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:27.603982 471 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet ddad7584337345109ce102702008ab37
I20250114 02:54:27.605732 471 catalog_manager.cc:6596] ReplaceTablet: tablet ddad7584337345109ce102702008ab37 deleted and replaced by tablet 7157b8c416d34c22b75953ae1994df33
I20250114 02:54:27.506099 2016 raft_consensus.cc:491] T 7774c2087422492696a58e0075921ae7 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:27.607699 2016 raft_consensus.cc:513] T 7774c2087422492696a58e0075921ae7 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:27.609984 2016 leader_election.cc:290] T 7774c2087422492696a58e0075921ae7 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:27.521449 932 tablet_service.cc:1467] Processing CreateTablet for tablet 53983c5af2444e419ccdd0be395bffa9 (DEFAULT_TABLE table=default.Table_b8047d6efb6845768a8d5192b794bd2a [id=24be7551945c40bc84b668b8747cf0e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:27.611091 932 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53983c5af2444e419ccdd0be395bffa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:27.499992 462 catalog_manager.cc:5526] T 42cee9e8e26147b593ed73fca6ac1183 P 525198593c8e47afaee54e74267b87e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 525198593c8e47afaee54e74267b87e3 (127.0.107.66). New cstate: current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:27.523402 938 tablet_service.cc:1467] Processing CreateTablet for tablet 587a5f39282a40d0a8c1591fdf36f341 (DEFAULT_TABLE table=default.Table_bacb81fcfeea434abb622c33872e1fde [id=8d818f58d351405f999498f48150fb48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:27.616914 938 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 587a5f39282a40d0a8c1591fdf36f341. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:27.619681 462 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet ce7d6088fc884a248918d2348adcaccf (table default.Table_428bd60f16824ac2b8bfaad6819f76e1 [id=35c5879da4b344118a377335fc6f83de]). Expected version 1 got 0
I20250114 02:54:27.527361 933 tablet_service.cc:1467] Processing CreateTablet for tablet e285f86c29654b9697427516e99ca43a (DEFAULT_TABLE table=default.Table_dd50844b27ef42dda7cab6f276fac825 [id=af4dd8face234ca99bf7523004217964]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:27.536896 941 tablet_service.cc:1467] Processing CreateTablet for tablet 63e7927546e04006b5533769b226fe00 (DEFAULT_TABLE table=default.Table_945582460ea34007972cb70eb35e608a [id=22dcb9796ed94aba85494dfd591d9542]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:27.621001 933 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e285f86c29654b9697427516e99ca43a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:27.626312 941 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63e7927546e04006b5533769b226fe00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:27.631105 940 tablet_service.cc:1467] Processing CreateTablet for tablet 8522b268234d41fd9bd3acd16a2add7c (DEFAULT_TABLE table=default.Table_8826c1ea171946ab9bc9b3572bf349c4 [id=9d6566ee69c34ebc8b3b638d55babe88]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:27.631954 940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8522b268234d41fd9bd3acd16a2add7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:27.638396 470 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_cec05793ad284a18a7da3557696f5bcb" } modify_external_catalogs: true
I20250114 02:54:27.639123 470 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_cec05793ad284a18a7da3557696f5bcb" } modify_external_catalogs: true
I20250114 02:54:27.645234 465 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_26478578c48e45af84e6b020c897a5f2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:27.647253 465 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_26478578c48e45af84e6b020c897a5f2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:27.649371 471 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_4be1140a3b21462e9be7fe47af15dd09"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:27.651412 471 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4be1140a3b21462e9be7fe47af15dd09 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250114 02:54:27.550426 898 leader_election.cc:343] T edc69b66bea54625a4b15867b75986b7 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:27.558748 1965 tablet_replica.cc:331] T d3fb28717c6f4844af84bb817fe7dfc4 P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:27.596979 935 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b8361cd538046e0bcb4aac821748f67 with delete_type TABLET_DATA_DELETED (replaced by tablet f0e6054998df447287fdff45a6504050) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:27.606861 942 tablet_service.cc:1514] Processing DeleteTablet for tablet ddad7584337345109ce102702008ab37 with delete_type TABLET_DATA_DELETED (replaced by tablet 7157b8c416d34c22b75953ae1994df33) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:27.671866 1205 raft_consensus.cc:383] T c86562688d5b4fffb71cbdf7be0afc37 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:27.672224 1205 raft_consensus.cc:738] T c86562688d5b4fffb71cbdf7be0afc37 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
W20250114 02:54:27.671442 898 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.67:45749, user_credentials={real_user=slave}} blocked reactor thread for 121047us
I20250114 02:54:27.695964 1965 raft_consensus.cc:2238] T d3fb28717c6f4844af84bb817fe7dfc4 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:27.697270 1965 raft_consensus.cc:2267] T d3fb28717c6f4844af84bb817fe7dfc4 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:27.699797 1965 ts_tablet_manager.cc:1905] T d3fb28717c6f4844af84bb817fe7dfc4 P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:27.705466 942 tablet_service.cc:1467] Processing CreateTablet for tablet e8ee0d1881eb4a9c8e6055317864d010 (DEFAULT_TABLE table=default.Table_96b6a43fc44a43ce9f6485f0e8daae5d [id=7e19cb37c53e4636941205a2d542984b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:27.706609 942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8ee0d1881eb4a9c8e6055317864d010. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:27.708647 935 tablet_service.cc:1467] Processing CreateTablet for tablet 39dfb04e741c4a859af39ddf7c691e0c (DEFAULT_TABLE table=default.Table_c778a0b532b64bcdbffa5cca153bcfcc [id=225e06c5be3042bab04da940d4caa996]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:27.709792 935 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39dfb04e741c4a859af39ddf7c691e0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:27.711306 462 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 92cd8eccacf34ab7a66b3bbc69df3697 (table default.Table_990b8d5f1bd146d6b358457f954780c9 [id=26627ccdc808451fb9593ad7a6df6991]). Expected version 2 got 0
I20250114 02:54:27.625173 788 tablet_service.cc:1467] Processing CreateTablet for tablet 63e7927546e04006b5533769b226fe00 (DEFAULT_TABLE table=default.Table_945582460ea34007972cb70eb35e608a [id=22dcb9796ed94aba85494dfd591d9542]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:27.721560 788 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63e7927546e04006b5533769b226fe00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:27.731773 462 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 044832851f804208b0a371eb0857dbf3 (table default.Table_0518d8bed7304c86aff32e05e6f07c0d [id=515a2fe3f1e84e82bccc02ef5cecef1b]). Expected version 1 got 0
I20250114 02:54:27.742234 471 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_6c885213c7cb45e2b12955642171daff" } new_table_name: "default.Table_0139fab1a81440309d9d7729c3e76c8a" modify_external_catalogs: true
I20250114 02:54:27.747148 796 tablet_service.cc:1514] Processing DeleteTablet for tablet ddad7584337345109ce102702008ab37 with delete_type TABLET_DATA_DELETED (replaced by tablet 7157b8c416d34c22b75953ae1994df33) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:27.743623 465 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_4ea70b8c32474894a62595dd9e0a1ae4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:27.747542 898 leader_election.cc:304] T edc69b66bea54625a4b15867b75986b7 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters: 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:27.748412 2016 raft_consensus.cc:2798] T edc69b66bea54625a4b15867b75986b7 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:27.748765 2016 raft_consensus.cc:491] T edc69b66bea54625a4b15867b75986b7 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:27.749315 2016 raft_consensus.cc:3054] T edc69b66bea54625a4b15867b75986b7 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:27.753444 934 tablet_service.cc:1467] Processing CreateTablet for tablet d0cc8b9e3aee4292bc69e8053d7e8cc1 (DEFAULT_TABLE table=default.Table_ee814dd266e64286864db0eac9e11664 [id=13151866bf2b4f44b1dd5c92380a4d43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:27.754746 934 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0cc8b9e3aee4292bc69e8053d7e8cc1. 1 dirs total, 0 dirs full, 0 dirs failed
W20250114 02:54:27.748751 465 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4ea70b8c32474894a62595dd9e0a1ae4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:27.749229 1205 consensus_queue.cc:260] T c86562688d5b4fffb71cbdf7be0afc37 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:27.763254 1961 raft_consensus.cc:491] T 6e2fb05f72b743e2a02ed36b63f80b23 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:27.764075 939 tablet_service.cc:1467] Processing CreateTablet for tablet 95ae9c2fb7c7489da29a48c6fd8c5fc1 (DEFAULT_TABLE table=default.Table_cd7d049881524116bf642f02443f44ad [id=c9ca7d97b22041d482cdd4ec5467c865]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:27.763811 1961 raft_consensus.cc:513] T 6e2fb05f72b743e2a02ed36b63f80b23 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:27.774204 817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7774c2087422492696a58e0075921ae7" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:27.774928 817 raft_consensus.cc:2463] T 7774c2087422492696a58e0075921ae7 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:27.777827 1961 leader_election.cc:290] T 6e2fb05f72b743e2a02ed36b63f80b23 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:27.793566 1205 ts_tablet_manager.cc:1428] T c86562688d5b4fffb71cbdf7be0afc37 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.298s user 0.005s sys 0.000s
I20250114 02:54:27.794710 1205 tablet_bootstrap.cc:492] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:27.514344 1103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99b940d872ce4781907517c40fe08781" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:27.519286 1207 tablet_bootstrap.cc:492] T 7774c2087422492696a58e0075921ae7 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:27.801071 1207 ts_tablet_manager.cc:1397] T 7774c2087422492696a58e0075921ae7 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.324s user 0.009s sys 0.000s
I20250114 02:54:27.803249 1207 raft_consensus.cc:357] T 7774c2087422492696a58e0075921ae7 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:27.803854 1207 raft_consensus.cc:383] T 7774c2087422492696a58e0075921ae7 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:27.804147 1207 raft_consensus.cc:738] T 7774c2087422492696a58e0075921ae7 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:27.805006 1207 consensus_queue.cc:260] T 7774c2087422492696a58e0075921ae7 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:27.807303 1207 ts_tablet_manager.cc:1428] T 7774c2087422492696a58e0075921ae7 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250114 02:54:27.551328 1101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99b940d872ce4781907517c40fe08781" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:27.808890 1207 tablet_bootstrap.cc:492] T edc69b66bea54625a4b15867b75986b7 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:27.576263 1990 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 247) took 2042 ms. Trace:
I20250114 02:54:27.632637 1966 raft_consensus.cc:491] T 84738a6c78b24c1fb64013048763b18f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:27.813115 1966 raft_consensus.cc:513] T 84738a6c78b24c1fb64013048763b18f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:27.814978 1966 leader_election.cc:290] T 84738a6c78b24c1fb64013048763b18f P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:27.815369 1990 rpcz_store.cc:276] 0114 02:54:25.533419 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:25.533516 (+ 97us) service_pool.cc:225] Handling call
0114 02:54:27.317998 (+1784482us) ts_tablet_manager.cc:1145] Deleting tablet ca986f81ed6848a89588366f3f400eec
0114 02:54:27.318046 (+ 48us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:27.382630 (+ 64584us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:27.423381 (+ 40751us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:27.576169 (+152788us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":168109,"tablet-delete.queue_time_us":1783967}
I20250114 02:54:27.816748 1990 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 248) took 2255 ms. Trace:
I20250114 02:54:27.817040 1990 rpcz_store.cc:276] 0114 02:54:25.561371 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:25.561555 (+ 184us) service_pool.cc:225] Handling call
0114 02:54:27.816466 (+2254911us) ts_tablet_manager.cc:1145] Deleting tablet 0f1ea1608213492c9a216757896ff2a4
0114 02:54:27.816517 (+ 51us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:27.816690 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2171142}
I20250114 02:54:27.818014 1990 tablet_replica.cc:331] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:27.818830 1990 raft_consensus.cc:2238] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:27.820294 1990 raft_consensus.cc:2267] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250114 02:54:27.820690 1042 leader_election.cc:343] T 84738a6c78b24c1fb64013048763b18f P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Not found: Tablet not found: 84738a6c78b24c1fb64013048763b18f
I20250114 02:54:27.826715 2058 raft_consensus.cc:491] T 38fe8829d98547fb907ef78099df3d7f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:27.827159 2058 raft_consensus.cc:513] T 38fe8829d98547fb907ef78099df3d7f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:27.828958 2058 leader_election.cc:290] T 38fe8829d98547fb907ef78099df3d7f P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:27.832733 958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e2fb05f72b743e2a02ed36b63f80b23" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
W20250114 02:54:27.844360 2088 log.cc:926] Time spent T 99b940d872ce4781907517c40fe08781 P 819f9e3412e54d44abb11d8b3ca877c2: Append to log took a long time: real 0.058s user 0.000s sys 0.001s
W20250114 02:54:27.786646 1040 leader_election.cc:343] T 5ac7708550954a209e2a8d217c3b39bd P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Not found: Tablet not found: 5ac7708550954a209e2a8d217c3b39bd
I20250114 02:54:27.788424 1104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7774c2087422492696a58e0075921ae7" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
W20250114 02:54:27.848110 1175 log.cc:926] Time spent T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Append to log took a long time: real 0.076s user 0.000s sys 0.001s
W20250114 02:54:27.847476 1181 log.cc:926] Time spent T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70: Append to log took a long time: real 0.068s user 0.000s sys 0.001s
I20250114 02:54:27.857905 939 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95ae9c2fb7c7489da29a48c6fd8c5fc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:27.863107 470 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet d56255ca806d4e2cac09fbb5e1b1efbd
I20250114 02:54:27.885397 1206 tablet_bootstrap.cc:492] T 59a46bc865cc48df8e98598764fd1b54 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:27.885799 1206 ts_tablet_manager.cc:1397] T 59a46bc865cc48df8e98598764fd1b54 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.412s user 0.025s sys 0.028s
I20250114 02:54:27.893494 465 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_e23da46545d64974b1f1563734b3c21c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:27.894922 465 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e23da46545d64974b1f1563734b3c21c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:27.845275 1040 leader_election.cc:304] T 5ac7708550954a209e2a8d217c3b39bd P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90d6c2c58bb54e0eafb365625ac33871; no voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2
I20250114 02:54:27.908428 1104 raft_consensus.cc:2463] T 7774c2087422492696a58e0075921ae7 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:27.913525 1207 tablet_bootstrap.cc:654] T edc69b66bea54625a4b15867b75986b7 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:27.921617 937 tablet_service.cc:1514] Processing DeleteTablet for tablet d56255ca806d4e2cac09fbb5e1b1efbd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:27 UTC) from {username='slave'} at 127.0.0.1:55068
W20250114 02:54:27.922153 1040 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.66:44037, user_credentials={real_user=slave}} blocked reactor thread for 135543us
I20250114 02:54:27.925022 465 master_service.cc:946] ReplaceTablet: received request to replace tablet 992bb16197b74be5b1dc884f29d9c240 from {username='slave'} at 127.0.0.1:33540
W20250114 02:54:27.926265 753 leader_election.cc:343] T 99b940d872ce4781907517c40fe08781 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:27.926779 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet ca986f81ed6848a89588366f3f400eec (table default.Table_123095518d164d3db7fe3bd50af9dbd4 [id=4744eda447ca4785a9c065aca94135e9]) successfully deleted
W20250114 02:54:27.927101 753 leader_election.cc:343] T 99b940d872ce4781907517c40fe08781 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:27.936300 1966 raft_consensus.cc:2743] T 5ac7708550954a209e2a8d217c3b39bd P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250114 02:54:27.939214 458 catalog_manager.cc:4839] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 0f1ea1608213492c9a216757896ff2a4 because the tablet was not found. No further retry: Not found: Tablet not found: 0f1ea1608213492c9a216757896ff2a4
I20250114 02:54:27.946892 1102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38fe8829d98547fb907ef78099df3d7f" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:27.952473 1990 ts_tablet_manager.cc:1905] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:27.954294 465 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet 992bb16197b74be5b1dc884f29d9c240
I20250114 02:54:27.959664 1103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e2fb05f72b743e2a02ed36b63f80b23" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:27.963790 1083 tablet_service.cc:1467] Processing CreateTablet for tablet 53983c5af2444e419ccdd0be395bffa9 (DEFAULT_TABLE table=default.Table_b8047d6efb6845768a8d5192b794bd2a [id=24be7551945c40bc84b668b8747cf0e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:27.964776 1083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53983c5af2444e419ccdd0be395bffa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:27.965116 465 catalog_manager.cc:6596] ReplaceTablet: tablet 992bb16197b74be5b1dc884f29d9c240 deleted and replaced by tablet b8f78115ca2e4d70a710e6f965bdf66a
I20250114 02:54:27.966141 465 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_174825b874fc410fba44e7a1335609d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:27.967566 465 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_174825b874fc410fba44e7a1335609d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:27.971747 817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38fe8829d98547fb907ef78099df3d7f" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:27.972329 817 raft_consensus.cc:2463] T 38fe8829d98547fb907ef78099df3d7f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:27.887907 1206 raft_consensus.cc:357] T 59a46bc865cc48df8e98598764fd1b54 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:27.980484 1206 raft_consensus.cc:383] T 59a46bc865cc48df8e98598764fd1b54 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:27.980748 1206 raft_consensus.cc:738] T 59a46bc865cc48df8e98598764fd1b54 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:27.981356 1084 tablet_service.cc:1467] Processing CreateTablet for tablet 587a5f39282a40d0a8c1591fdf36f341 (DEFAULT_TABLE table=default.Table_bacb81fcfeea434abb622c33872e1fde [id=8d818f58d351405f999498f48150fb48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:27.981258 1206 consensus_queue.cc:260] T 59a46bc865cc48df8e98598764fd1b54 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:27.982321 1084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 587a5f39282a40d0a8c1591fdf36f341. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:27.987566 1992 ts_tablet_manager.cc:1918] T 46360f726fad4534a2daf67581346fc1 P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:27.994047 1086 tablet_service.cc:1467] Processing CreateTablet for tablet e285f86c29654b9697427516e99ca43a (DEFAULT_TABLE table=default.Table_dd50844b27ef42dda7cab6f276fac825 [id=af4dd8face234ca99bf7523004217964]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:27.994514 1085 tablet_service.cc:1467] Processing CreateTablet for tablet 63e7927546e04006b5533769b226fe00 (DEFAULT_TABLE table=default.Table_945582460ea34007972cb70eb35e608a [id=22dcb9796ed94aba85494dfd591d9542]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:27.995157 1086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e285f86c29654b9697427516e99ca43a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:27.994514 1080 tablet_service.cc:1467] Processing CreateTablet for tablet 8522b268234d41fd9bd3acd16a2add7c (DEFAULT_TABLE table=default.Table_8826c1ea171946ab9bc9b3572bf349c4 [id=9d6566ee69c34ebc8b3b638d55babe88]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:27.996429 1081 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b8361cd538046e0bcb4aac821748f67 with delete_type TABLET_DATA_DELETED (replaced by tablet f0e6054998df447287fdff45a6504050) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:27.996505 1080 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8522b268234d41fd9bd3acd16a2add7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:27.997510 1082 tablet_service.cc:1514] Processing DeleteTablet for tablet ddad7584337345109ce102702008ab37 with delete_type TABLET_DATA_DELETED (replaced by tablet 7157b8c416d34c22b75953ae1994df33) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:28.000478 1085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63e7927546e04006b5533769b226fe00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:28.001984 1207 tablet_bootstrap.cc:492] T edc69b66bea54625a4b15867b75986b7 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:28.002326 1207 ts_tablet_manager.cc:1397] T edc69b66bea54625a4b15867b75986b7 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.194s user 0.009s sys 0.008s
I20250114 02:54:28.005295 465 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_23e9951e382b44538252984608700dcc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:28.006843 465 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_23e9951e382b44538252984608700dcc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:28.008112 1207 raft_consensus.cc:357] T edc69b66bea54625a4b15867b75986b7 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:28.008782 1207 raft_consensus.cc:383] T edc69b66bea54625a4b15867b75986b7 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:28.009025 1207 raft_consensus.cc:738] T edc69b66bea54625a4b15867b75986b7 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:28.017262 1079 tablet_service.cc:1467] Processing CreateTablet for tablet e8ee0d1881eb4a9c8e6055317864d010 (DEFAULT_TABLE table=default.Table_96b6a43fc44a43ce9f6485f0e8daae5d [id=7e19cb37c53e4636941205a2d542984b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:28.018332 1079 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8ee0d1881eb4a9c8e6055317864d010. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:28.030107 1078 tablet_service.cc:1514] Processing DeleteTablet for tablet d56255ca806d4e2cac09fbb5e1b1efbd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:27 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:28.029935 1082 tablet_service.cc:1467] Processing CreateTablet for tablet d0cc8b9e3aee4292bc69e8053d7e8cc1 (DEFAULT_TABLE table=default.Table_ee814dd266e64286864db0eac9e11664 [id=13151866bf2b4f44b1dd5c92380a4d43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:28.030949 1081 tablet_service.cc:1467] Processing CreateTablet for tablet 95ae9c2fb7c7489da29a48c6fd8c5fc1 (DEFAULT_TABLE table=default.Table_cd7d049881524116bf642f02443f44ad [id=c9ca7d97b22041d482cdd4ec5467c865]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:28.031234 1082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0cc8b9e3aee4292bc69e8053d7e8cc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:28.031832 1081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95ae9c2fb7c7489da29a48c6fd8c5fc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:28.050945 465 master_service.cc:946] ReplaceTablet: received request to replace tablet 59a46bc865cc48df8e98598764fd1b54 from {username='slave'} at 127.0.0.1:33540
I20250114 02:54:28.009557 1207 consensus_queue.cc:260] T edc69b66bea54625a4b15867b75986b7 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:28.067605 1207 ts_tablet_manager.cc:1428] T edc69b66bea54625a4b15867b75986b7 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.065s user 0.006s sys 0.000s
I20250114 02:54:28.068619 1207 tablet_bootstrap.cc:492] T 6e858a8449444ef999b9bbef612249d3 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:28.072875 1084 tablet_service.cc:1467] Processing CreateTablet for tablet 39dfb04e741c4a859af39ddf7c691e0c (DEFAULT_TABLE table=default.Table_c778a0b532b64bcdbffa5cca153bcfcc [id=225e06c5be3042bab04da940d4caa996]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:28.074029 1084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39dfb04e741c4a859af39ddf7c691e0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:28.084826 1083 tablet_service.cc:1514] Processing DeleteTablet for tablet 992bb16197b74be5b1dc884f29d9c240 with delete_type TABLET_DATA_DELETED (replaced by tablet b8f78115ca2e4d70a710e6f965bdf66a) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:28.096455 898 leader_election.cc:304] T 7774c2087422492696a58e0075921ae7 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
W20250114 02:54:28.075563 2087 log.cc:926] Time spent T ddfbab181b1f474192c664cd0316037f P 525198593c8e47afaee54e74267b87e3: Append to log took a long time: real 0.342s user 0.000s sys 0.000s
I20250114 02:54:28.048280 2058 raft_consensus.cc:491] T 719188f639724851a5fc2b37cb60fdbb P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:28.114938 942 tablet_service.cc:1514] Processing DeleteTablet for tablet 992bb16197b74be5b1dc884f29d9c240 with delete_type TABLET_DATA_DELETED (replaced by tablet b8f78115ca2e4d70a710e6f965bdf66a) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:28.106231 2097 raft_consensus.cc:2798] T 7774c2087422492696a58e0075921ae7 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:28.119015 2097 raft_consensus.cc:491] T 7774c2087422492696a58e0075921ae7 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:28.119670 2097 raft_consensus.cc:3054] T 7774c2087422492696a58e0075921ae7 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:28.125591 1207 tablet_bootstrap.cc:654] T 6e858a8449444ef999b9bbef612249d3 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:28.146968 1207 tablet_bootstrap.cc:492] T 6e858a8449444ef999b9bbef612249d3 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:28.147419 1207 ts_tablet_manager.cc:1397] T 6e858a8449444ef999b9bbef612249d3 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.079s user 0.007s sys 0.002s
I20250114 02:54:28.157855 1207 raft_consensus.cc:357] T 6e858a8449444ef999b9bbef612249d3 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:28.158551 1207 raft_consensus.cc:383] T 6e858a8449444ef999b9bbef612249d3 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:28.158810 1207 raft_consensus.cc:738] T 6e858a8449444ef999b9bbef612249d3 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:28.160373 1207 consensus_queue.cc:260] T 6e858a8449444ef999b9bbef612249d3 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:28.165315 1207 ts_tablet_manager.cc:1428] T 6e858a8449444ef999b9bbef612249d3 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.018s user 0.006s sys 0.000s
I20250114 02:54:28.168996 1207 tablet_bootstrap.cc:492] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:28.192797 2016 raft_consensus.cc:513] T edc69b66bea54625a4b15867b75986b7 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:28.205437 1207 tablet_bootstrap.cc:654] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:28.149786 1965 ts_tablet_manager.cc:1918] T d3fb28717c6f4844af84bb817fe7dfc4 P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:28.218874 1965 log.cc:1198] T d3fb28717c6f4844af84bb817fe7dfc4 P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/d3fb28717c6f4844af84bb817fe7dfc4
I20250114 02:54:28.219920 1965 ts_tablet_manager.cc:1939] T d3fb28717c6f4844af84bb817fe7dfc4 P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:28.228493 1966 raft_consensus.cc:491] T 6e858a8449444ef999b9bbef612249d3 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:28.228922 1966 raft_consensus.cc:513] T 6e858a8449444ef999b9bbef612249d3 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:28.230688 1966 leader_election.cc:290] T 6e858a8449444ef999b9bbef612249d3 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
W20250114 02:54:28.233214 456 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet ed7f575653c848d7b076c17c38c4a244 (table default.Table_f842a61f536940588cd46bedc206f2b3 [id=954669573a3c4a97a488805e05b0b220], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet ed7f575653c848d7b076c17c38c4a244 consensus state has no leader
I20250114 02:54:28.249574 2084 tablet.cc:1721] T ce7d6088fc884a248918d2348adcaccf P 90d6c2c58bb54e0eafb365625ac33871: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250114 02:54:28.224639 2016 leader_election.cc:290] T edc69b66bea54625a4b15867b75986b7 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Requested vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:28.106870 2058 raft_consensus.cc:513] T 719188f639724851a5fc2b37cb60fdbb P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:28.292585 2058 leader_election.cc:290] T 719188f639724851a5fc2b37cb60fdbb P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
W20250114 02:54:28.309445 455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.107.124:39105: connect: Connection refused (error 111) [suppressed 4 similar messages]
I20250114 02:54:28.315939 1207 tablet_bootstrap.cc:492] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:28.316408 1207 ts_tablet_manager.cc:1397] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.148s user 0.026s sys 0.022s
I20250114 02:54:28.317214 959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e858a8449444ef999b9bbef612249d3" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:28.016010 790 tablet_service.cc:1467] Processing CreateTablet for tablet d0cc8b9e3aee4292bc69e8053d7e8cc1 (DEFAULT_TABLE table=default.Table_ee814dd266e64286864db0eac9e11664 [id=13151866bf2b4f44b1dd5c92380a4d43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:28.318655 1992 log.cc:1198] T 46360f726fad4534a2daf67581346fc1 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/46360f726fad4534a2daf67581346fc1
I20250114 02:54:28.319262 790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0cc8b9e3aee4292bc69e8053d7e8cc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:28.319466 1992 ts_tablet_manager.cc:1939] T 46360f726fad4534a2daf67581346fc1 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:28.320309 1992 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 230) took 2669 ms. Trace:
I20250114 02:54:28.320632 1992 rpcz_store.cc:276] 0114 02:54:25.650461 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:25.650584 (+ 123us) service_pool.cc:225] Handling call
0114 02:54:27.179482 (+1528898us) ts_tablet_manager.cc:1145] Deleting tablet 46360f726fad4534a2daf67581346fc1
0114 02:54:27.179528 (+ 46us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:27.882414 (+702886us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:27.987346 (+104932us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:28.320210 (+332864us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1528191}
I20250114 02:54:28.321913 1992 tablet_replica.cc:331] T 1381e26725174b9c877075cf5444f566 P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:28.322798 1992 raft_consensus.cc:2238] T 1381e26725174b9c877075cf5444f566 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:28.323343 1992 raft_consensus.cc:2267] T 1381e26725174b9c877075cf5444f566 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:28.324611 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet 46360f726fad4534a2daf67581346fc1 (table default.Table_3e7cad29b0c0447ab9539b45ee482232 [id=f65443bbd53c47e184a6d2e9e99dd1b0]) successfully deleted
I20250114 02:54:28.324721 1965 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55068 (request call id 268) took 1784 ms. Trace:
I20250114 02:54:28.023319 797 tablet_service.cc:1467] Processing CreateTablet for tablet 39dfb04e741c4a859af39ddf7c691e0c (DEFAULT_TABLE table=default.Table_c778a0b532b64bcdbffa5cca153bcfcc [id=225e06c5be3042bab04da940d4caa996]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:28.325017 1965 rpcz_store.cc:276] 0114 02:54:26.539904 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:26.540031 (+ 127us) service_pool.cc:225] Handling call
0114 02:54:27.474421 (+934390us) ts_tablet_manager.cc:1145] Deleting tablet d3fb28717c6f4844af84bb817fe7dfc4
0114 02:54:27.553559 (+ 79138us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:27.873414 (+319855us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:28.149559 (+276145us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:28.324661 (+175102us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":766,"tablet-delete.queue_time_us":925998}
I20250114 02:54:28.325899 797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39dfb04e741c4a859af39ddf7c691e0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:28.326243 1965 tablet_replica.cc:331] T de4ca83a69704a368a25954cdc1b9392 P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:28.326968 1965 raft_consensus.cc:2238] T de4ca83a69704a368a25954cdc1b9392 P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Raft consensus shutting down.
I20250114 02:54:28.327667 1965 raft_consensus.cc:2267] T de4ca83a69704a368a25954cdc1b9392 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:28.056984 796 tablet_service.cc:1467] Processing CreateTablet for tablet 95ae9c2fb7c7489da29a48c6fd8c5fc1 (DEFAULT_TABLE table=default.Table_cd7d049881524116bf642f02443f44ad [id=c9ca7d97b22041d482cdd4ec5467c865]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:28.328523 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet d3fb28717c6f4844af84bb817fe7dfc4 (table default.Table_b8b1e6d9bd0742f7880a7f55e986d0ee [id=871e09c142b94b79aeff9bfd8d78e2bb]) successfully deleted
I20250114 02:54:28.328842 796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95ae9c2fb7c7489da29a48c6fd8c5fc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:28.057976 793 tablet_service.cc:1514] Processing DeleteTablet for tablet d56255ca806d4e2cac09fbb5e1b1efbd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:27 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:28.058435 792 tablet_service.cc:1514] Processing DeleteTablet for tablet 992bb16197b74be5b1dc884f29d9c240 with delete_type TABLET_DATA_DELETED (replaced by tablet b8f78115ca2e4d70a710e6f965bdf66a) from {username='slave'} at 127.0.0.1:53078
W20250114 02:54:28.334409 1040 consensus_peers.cc:487] T 46360f726fad4534a2daf67581346fc1 P 90d6c2c58bb54e0eafb365625ac33871 -> Peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Couldn't send request to peer 525198593c8e47afaee54e74267b87e3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 46360f726fad4534a2daf67581346fc1. This is attempt 1: this message will repeat every 5th retry.
I20250114 02:54:28.061100 789 tablet_service.cc:1467] Processing CreateTablet for tablet e8ee0d1881eb4a9c8e6055317864d010 (DEFAULT_TABLE table=default.Table_96b6a43fc44a43ce9f6485f0e8daae5d [id=7e19cb37c53e4636941205a2d542984b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:28.337608 789 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8ee0d1881eb4a9c8e6055317864d010. 1 dirs total, 0 dirs full, 0 dirs failed
W20250114 02:54:28.345050 1040 leader_election.cc:343] T 6e858a8449444ef999b9bbef612249d3 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:28.359120 1207 raft_consensus.cc:357] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:28.360055 1207 raft_consensus.cc:383] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:28.360402 1207 raft_consensus.cc:738] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:28.361024 1207 consensus_queue.cc:260] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:28.361501 2113 tablet.cc:1721] T ce7d6088fc884a248918d2348adcaccf P 525198593c8e47afaee54e74267b87e3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250114 02:54:28.240172 898 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.65:39397, user_credentials={real_user=slave}} blocked reactor thread for 142919us
I20250114 02:54:28.364674 898 leader_election.cc:304] T 38fe8829d98547fb907ef78099df3d7f P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
W20250114 02:54:28.366179 751 leader_election.cc:343] T 6e2fb05f72b743e2a02ed36b63f80b23 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:28.114271 788 tablet_service.cc:1467] Processing CreateTablet for tablet 8522b268234d41fd9bd3acd16a2add7c (DEFAULT_TABLE table=default.Table_8826c1ea171946ab9bc9b3572bf349c4 [id=9d6566ee69c34ebc8b3b638d55babe88]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:28.173331 1741 raft_consensus.cc:491] T 719188f639724851a5fc2b37cb60fdbb P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:28.369037 788 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8522b268234d41fd9bd3acd16a2add7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:28.368742 1741 raft_consensus.cc:513] T 719188f639724851a5fc2b37cb60fdbb P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:28.370606 1741 leader_election.cc:290] T 719188f639724851a5fc2b37cb60fdbb P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:28.254496 1205 tablet_bootstrap.cc:654] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:28.261420 958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84738a6c78b24c1fb64013048763b18f" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:28.380105 958 raft_consensus.cc:2463] T 84738a6c78b24c1fb64013048763b18f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 0.
I20250114 02:54:28.175263 1206 ts_tablet_manager.cc:1428] T 59a46bc865cc48df8e98598764fd1b54 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.289s user 0.006s sys 0.000s
W20250114 02:54:28.177968 753 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.67:45749, user_credentials={real_user=slave}} blocked reactor thread for 114012us
W20250114 02:54:28.385485 753 leader_election.cc:343] T 6e2fb05f72b743e2a02ed36b63f80b23 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:28.385848 753 leader_election.cc:304] T 6e2fb05f72b743e2a02ed36b63f80b23 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:28.232380 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e858a8449444ef999b9bbef612249d3" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:28.386969 818 raft_consensus.cc:2463] T 6e858a8449444ef999b9bbef612249d3 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 0.
I20250114 02:54:28.387814 2090 raft_consensus.cc:491] T c86562688d5b4fffb71cbdf7be0afc37 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:28.388576 1207 ts_tablet_manager.cc:1428] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.072s user 0.006s sys 0.000s
I20250114 02:54:28.388285 2090 raft_consensus.cc:513] T c86562688d5b4fffb71cbdf7be0afc37 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:28.389566 1207 tablet_bootstrap.cc:492] T d05089eae2b74f9fb23a226d358a59da P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:28.390015 2090 leader_election.cc:290] T c86562688d5b4fffb71cbdf7be0afc37 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:28.391673 2090 raft_consensus.cc:2743] T 6e2fb05f72b743e2a02ed36b63f80b23 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:28.392793 1961 raft_consensus.cc:491] T 59a46bc865cc48df8e98598764fd1b54 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:28.393177 1961 raft_consensus.cc:513] T 59a46bc865cc48df8e98598764fd1b54 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:28.394780 1961 leader_election.cc:290] T 59a46bc865cc48df8e98598764fd1b54 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:28.402925 1040 leader_election.cc:304] T 84738a6c78b24c1fb64013048763b18f P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters: 819f9e3412e54d44abb11d8b3ca877c2
I20250114 02:54:28.406559 1102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "719188f639724851a5fc2b37cb60fdbb" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:28.407827 1966 raft_consensus.cc:2798] T 84738a6c78b24c1fb64013048763b18f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:28.408164 1966 raft_consensus.cc:491] T 84738a6c78b24c1fb64013048763b18f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:28.408489 1966 raft_consensus.cc:3054] T 84738a6c78b24c1fb64013048763b18f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:28.412062 1207 tablet_bootstrap.cc:654] T d05089eae2b74f9fb23a226d358a59da P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:28.412690 1102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c86562688d5b4fffb71cbdf7be0afc37" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:28.415290 1103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59a46bc865cc48df8e98598764fd1b54" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:28.418097 1042 leader_election.cc:304] T 6e858a8449444ef999b9bbef612249d3 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters: 525198593c8e47afaee54e74267b87e3
I20250114 02:54:28.418892 2031 raft_consensus.cc:2798] T 6e858a8449444ef999b9bbef612249d3 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:28.419241 2031 raft_consensus.cc:491] T 6e858a8449444ef999b9bbef612249d3 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:28.419577 2031 raft_consensus.cc:3054] T 6e858a8449444ef999b9bbef612249d3 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
W20250114 02:54:28.425328 753 leader_election.cc:343] T 719188f639724851a5fc2b37cb60fdbb P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
W20250114 02:54:28.425967 753 leader_election.cc:343] T c86562688d5b4fffb71cbdf7be0afc37 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
W20250114 02:54:28.427691 753 leader_election.cc:343] T 59a46bc865cc48df8e98598764fd1b54 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:28.431114 2016 raft_consensus.cc:2798] T 38fe8829d98547fb907ef78099df3d7f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:28.431557 2016 raft_consensus.cc:491] T 38fe8829d98547fb907ef78099df3d7f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:28.432003 2016 raft_consensus.cc:3054] T 38fe8829d98547fb907ef78099df3d7f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
W20250114 02:54:28.440007 1206 ts_tablet_manager.cc:1460] T 59a46bc865cc48df8e98598764fd1b54 P 819f9e3412e54d44abb11d8b3ca877c2: Tablet startup took 553ms
W20250114 02:54:28.444737 1206 ts_tablet_manager.cc:1462] T 59a46bc865cc48df8e98598764fd1b54 P 819f9e3412e54d44abb11d8b3ca877c2: Trace:
0114 02:54:19.921328 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:19.921601 (+ 273us) service_pool.cc:225] Handling call
0114 02:54:20.027813 (+106212us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0114 02:54:20.027922 (+ 109us) ts_tablet_manager.cc:676] Creating new metadata...
0114 02:54:20.063700 (+ 35778us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:20.122962 (+ 59262us) tablet_replica.cc:206] Creating consensus instance
0114 02:54:20.123476 (+ 514us) inbound_call.cc:173] Queueing success response
0114 02:54:27.473593 (+7350117us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0114 02:54:27.886043 (+412450us) ts_tablet_manager.cc:1429] Starting tablet replica
0114 02:54:27.886469 (+ 426us) tablet_replica.cc:260] Starting instrumentation
0114 02:54:27.886907 (+ 438us) tablet_replica.cc:279] Starting consensus
Metrics: {"mutex_wait_us":22,"raft.queue_time_us":193627,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":181,"tablet-open.queue_time_us":7350224}
I20250114 02:54:28.450935 1966 raft_consensus.cc:513] T 84738a6c78b24c1fb64013048763b18f P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:28.463326 1206 tablet_bootstrap.cc:492] T ed7f575653c848d7b076c17c38c4a244 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:28.275427 2109 tablet.cc:1721] T ce7d6088fc884a248918d2348adcaccf P 819f9e3412e54d44abb11d8b3ca877c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250114 02:54:28.485551 1205 tablet_bootstrap.cc:492] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:28.485939 1205 ts_tablet_manager.cc:1397] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.691s user 0.006s sys 0.005s
I20250114 02:54:28.494832 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc69b66bea54625a4b15867b75986b7" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:28.495493 818 raft_consensus.cc:3054] T edc69b66bea54625a4b15867b75986b7 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:28.509305 818 raft_consensus.cc:2463] T edc69b66bea54625a4b15867b75986b7 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:28.515101 814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "719188f639724851a5fc2b37cb60fdbb" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:28.515748 814 raft_consensus.cc:2463] T 719188f639724851a5fc2b37cb60fdbb P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:28.238564 2097 raft_consensus.cc:513] T 7774c2087422492696a58e0075921ae7 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:28.530647 2097 leader_election.cc:290] T 7774c2087422492696a58e0075921ae7 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Requested vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
W20250114 02:54:28.537269 753 consensus_peers.cc:487] T 99b940d872ce4781907517c40fe08781 P 819f9e3412e54d44abb11d8b3ca877c2 -> Peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Couldn't send request to peer 90d6c2c58bb54e0eafb365625ac33871. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250114 02:54:28.548130 2119 raft_consensus.cc:491] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b303b7df086340a0b5ad6373940ae750)
I20250114 02:54:28.552181 2119 raft_consensus.cc:513] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b303b7df086340a0b5ad6373940ae750" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 45367 } } peers { permanent_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } } peers { permanent_uuid: "87b3db09ac87436e9a60d6de5fedaff5" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } }
I20250114 02:54:28.555310 2119 leader_election.cc:290] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b303b7df086340a0b5ad6373940ae750 (127.0.107.126:45367), 87b3db09ac87436e9a60d6de5fedaff5 (127.0.107.124:39105)
I20250114 02:54:28.555111 481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" candidate_term: 2 candidate_status { last_received { term: 1 index: 364 } } ignore_live_leader: false dest_uuid: "b303b7df086340a0b5ad6373940ae750" is_pre_election: true
W20250114 02:54:28.562004 530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.107.124:39105: connect: Connection refused (error 111)
I20250114 02:54:28.573606 1205 raft_consensus.cc:357] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:28.574293 1205 raft_consensus.cc:383] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:28.574597 1205 raft_consensus.cc:738] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:28.575280 2105 raft_consensus.cc:491] T c86562688d5b4fffb71cbdf7be0afc37 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:28.575137 1205 consensus_queue.cc:260] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:28.575675 2105 raft_consensus.cc:513] T c86562688d5b4fffb71cbdf7be0afc37 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:28.576805 1205 ts_tablet_manager.cc:1428] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.091s user 0.005s sys 0.002s
I20250114 02:54:28.577378 2105 leader_election.cc:290] T c86562688d5b4fffb71cbdf7be0afc37 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:28.577821 1205 tablet_bootstrap.cc:492] T df045d80d8ce4ff194a6d2039a44bce6 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:28.583551 1205 tablet_bootstrap.cc:654] T df045d80d8ce4ff194a6d2039a44bce6 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:28.606590 1965 ts_tablet_manager.cc:1905] T de4ca83a69704a368a25954cdc1b9392 P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:28.439023 2031 raft_consensus.cc:513] T 6e858a8449444ef999b9bbef612249d3 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:28.479123 1103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc69b66bea54625a4b15867b75986b7" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:28.484382 1104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "719188f639724851a5fc2b37cb60fdbb" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:28.535305 1820 raft_consensus.cc:491] T edc69b66bea54625a4b15867b75986b7 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 02:54:28.612423 530 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.124:39105, user_credentials={real_user=slave}} blocked reactor thread for 50481.9us
W20250114 02:54:28.614821 530 leader_election.cc:336] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 87b3db09ac87436e9a60d6de5fedaff5 (127.0.107.124:39105): Network error: Client connection negotiation failed: client connection to 127.0.107.124:39105: connect: Connection refused (error 111)
I20250114 02:54:28.615391 530 leader_election.cc:304] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 687d4ff6d3cb42cf90a965070b0d8a70; no voters: 87b3db09ac87436e9a60d6de5fedaff5, b303b7df086340a0b5ad6373940ae750
I20250114 02:54:28.612622 1820 raft_consensus.cc:513] T edc69b66bea54625a4b15867b75986b7 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:28.616319 2119 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:54:28.616701 1103 raft_consensus.cc:3054] T edc69b66bea54625a4b15867b75986b7 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:28.621277 1820 leader_election.cc:290] T edc69b66bea54625a4b15867b75986b7 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:28.624650 1103 raft_consensus.cc:2463] T edc69b66bea54625a4b15867b75986b7 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:28.613848 2031 leader_election.cc:290] T 6e858a8449444ef999b9bbef612249d3 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Requested vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
W20250114 02:54:28.605031 1042 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.65:39397, user_credentials={real_user=slave}} blocked reactor thread for 54040.9us
I20250114 02:54:28.645355 788 tablet_service.cc:1467] Processing CreateTablet for tablet 4ab79e0a576843e89a2d524608c5b7f4 (DEFAULT_TABLE table=default.Table_4be1140a3b21462e9be7fe47af15dd09 [id=5496cdb5c870491dbf3373b3c6cd927f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:28.646345 788 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ab79e0a576843e89a2d524608c5b7f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:28.649896 468 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet edc69b66bea54625a4b15867b75986b7 (table default.Table_0a0bda55a4a9476c98d961822b8a19bd [id=fb2be5c1ca9444a197f3315ea9c59b09]). Expected version 1 got 0
I20250114 02:54:28.651952 468 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet ce7d6088fc884a248918d2348adcaccf (table default.Table_428bd60f16824ac2b8bfaad6819f76e1 [id=35c5879da4b344118a377335fc6f83de]). Expected version 1 got 0
I20250114 02:54:28.658654 789 tablet_service.cc:1467] Processing CreateTablet for tablet 5fa22baf436f4815a136673b12a476c7 (DEFAULT_TABLE table=default.Table_42771479a01b45f7ade52d2ec3896c4a [id=312b160396c64566981a3b6325a8778d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:28.659678 789 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fa22baf436f4815a136673b12a476c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:28.660838 790 tablet_service.cc:1467] Processing CreateTablet for tablet 6cf7f6a9debe4ba3bb0b58fc5f48980c (DEFAULT_TABLE table=default.Table_8d2e7b6869da423a949c59081ea3618d [id=9cf2e181b6bc44f3b96312c76ba6e646]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:28.661742 790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6cf7f6a9debe4ba3bb0b58fc5f48980c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:28.664072 817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e858a8449444ef999b9bbef612249d3" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:28.664592 817 raft_consensus.cc:3054] T 6e858a8449444ef999b9bbef612249d3 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:28.664786 797 tablet_service.cc:1467] Processing CreateTablet for tablet 9da1c960cb3d4b57869a307afdb41ac3 (DEFAULT_TABLE table=default.Table_3cd1350ab3034d69b3f85470c9ab3e34 [id=ed3b05cec0a348a5badacea3e6c47ee6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:28.665634 797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9da1c960cb3d4b57869a307afdb41ac3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:28.666471 1966 leader_election.cc:290] T 84738a6c78b24c1fb64013048763b18f P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Requested vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:28.669683 814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc69b66bea54625a4b15867b75986b7" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:28.670393 814 raft_consensus.cc:2388] T edc69b66bea54625a4b15867b75986b7 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 90d6c2c58bb54e0eafb365625ac33871 in current term 1: Already voted for candidate 525198593c8e47afaee54e74267b87e3 in this term.
I20250114 02:54:28.611017 959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84738a6c78b24c1fb64013048763b18f" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
I20250114 02:54:28.672583 959 raft_consensus.cc:3054] T 84738a6c78b24c1fb64013048763b18f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:28.614754 957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e858a8449444ef999b9bbef612249d3" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
I20250114 02:54:28.630829 960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc69b66bea54625a4b15867b75986b7" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:28.677302 960 raft_consensus.cc:2388] T edc69b66bea54625a4b15867b75986b7 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 90d6c2c58bb54e0eafb365625ac33871 in current term 1: Already voted for candidate 525198593c8e47afaee54e74267b87e3 in this term.
I20250114 02:54:28.677148 796 tablet_service.cc:1467] Processing CreateTablet for tablet d8d16aad0fed4f44b4a479a3e4f77bc9 (DEFAULT_TABLE table=default.Table_26355cc977e54afca705ed400dae8207 [id=0436dc9f10d74e7eb434ac41f704af39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:28.640988 934 tablet_service.cc:1467] Processing CreateTablet for tablet 4ab79e0a576843e89a2d524608c5b7f4 (DEFAULT_TABLE table=default.Table_4be1140a3b21462e9be7fe47af15dd09 [id=5496cdb5c870491dbf3373b3c6cd927f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:28.678057 796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8d16aad0fed4f44b4a479a3e4f77bc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:28.678419 934 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ab79e0a576843e89a2d524608c5b7f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:28.645417 935 tablet_service.cc:1467] Processing CreateTablet for tablet 5fa22baf436f4815a136673b12a476c7 (DEFAULT_TABLE table=default.Table_42771479a01b45f7ade52d2ec3896c4a [id=312b160396c64566981a3b6325a8778d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:28.680891 935 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fa22baf436f4815a136673b12a476c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:28.680895 468 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 38fe8829d98547fb907ef78099df3d7f (table default.Table_60450873174a4c80aa7e2122343d82be [id=9299683349354f879847b7c9d93f6072]). Expected version 1 got 0
I20250114 02:54:28.681512 468 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 044832851f804208b0a371eb0857dbf3 (table default.Table_0518d8bed7304c86aff32e05e6f07c0d [id=515a2fe3f1e84e82bccc02ef5cecef1b]). Expected version 1 got 0
I20250114 02:54:28.682256 468 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 92cd8eccacf34ab7a66b3bbc69df3697 (table default.Table_990b8d5f1bd146d6b358457f954780c9 [id=26627ccdc808451fb9593ad7a6df6991]). Expected version 2 got 0
I20250114 02:54:28.682842 468 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet a52d6749ff834b62a8e20a2f45935240 (table default.Table_1c9b7896941e4289b1c12e20c210c2c1 [id=b3ee15b9f10f4d489edbde3af9034d25]). Expected version 1 got 0
I20250114 02:54:28.682324 792 tablet_service.cc:1467] Processing CreateTablet for tablet 25bce689024f49a49496e415efe91fdc (DEFAULT_TABLE table=default.Table_e18d67a6f19d4259bee6f447deced18c [id=b3d24ad0e5b7449d808b8b1826fc07c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:28.683691 792 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25bce689024f49a49496e415efe91fdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:28.685122 793 tablet_service.cc:1467] Processing CreateTablet for tablet f414957450cc49fdb59776e6fd8acb5d (DEFAULT_TABLE table=default.Table_6071c0da3a0d4ecc9e063e0bab9712de [id=226c86aee94b4aa3b1569e2d4784e842]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:28.686014 793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f414957450cc49fdb59776e6fd8acb5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:28.695348 464 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet 4bb9b429af92446bbe13894d5dfe4c47 (table default.Table_e4cc8164591444a4bf1e683f3a609d36 [id=22b7edcf4b7e4c31b3c23dc44ccf028e]). Expected version 2 got 1
I20250114 02:54:28.698660 959 raft_consensus.cc:2463] T 84738a6c78b24c1fb64013048763b18f P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1.
I20250114 02:54:28.655072 939 tablet_service.cc:1467] Processing CreateTablet for tablet 6cf7f6a9debe4ba3bb0b58fc5f48980c (DEFAULT_TABLE table=default.Table_8d2e7b6869da423a949c59081ea3618d [id=9cf2e181b6bc44f3b96312c76ba6e646]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:28.656955 938 tablet_service.cc:1467] Processing CreateTablet for tablet 9da1c960cb3d4b57869a307afdb41ac3 (DEFAULT_TABLE table=default.Table_3cd1350ab3034d69b3f85470c9ab3e34 [id=ed3b05cec0a348a5badacea3e6c47ee6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:28.700356 939 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6cf7f6a9debe4ba3bb0b58fc5f48980c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:28.700908 938 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9da1c960cb3d4b57869a307afdb41ac3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:28.702265 941 tablet_service.cc:1467] Processing CreateTablet for tablet 25bce689024f49a49496e415efe91fdc (DEFAULT_TABLE table=default.Table_e18d67a6f19d4259bee6f447deced18c [id=b3d24ad0e5b7449d808b8b1826fc07c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:28.703099 941 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25bce689024f49a49496e415efe91fdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:28.704865 942 tablet_service.cc:1467] Processing CreateTablet for tablet d8d16aad0fed4f44b4a479a3e4f77bc9 (DEFAULT_TABLE table=default.Table_26355cc977e54afca705ed400dae8207 [id=0436dc9f10d74e7eb434ac41f704af39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:28.705749 942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8d16aad0fed4f44b4a479a3e4f77bc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:28.717391 932 tablet_service.cc:1467] Processing CreateTablet for tablet f414957450cc49fdb59776e6fd8acb5d (DEFAULT_TABLE table=default.Table_6071c0da3a0d4ecc9e063e0bab9712de [id=226c86aee94b4aa3b1569e2d4784e842]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:28.718436 932 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f414957450cc49fdb59776e6fd8acb5d. 1 dirs total, 0 dirs full, 0 dirs failed
W20250114 02:54:28.720839 1040 leader_election.cc:343] T 6e858a8449444ef999b9bbef612249d3 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:28.721870 1040 leader_election.cc:304] T 84738a6c78b24c1fb64013048763b18f P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:28.723659 1084 tablet_service.cc:1467] Processing CreateTablet for tablet 5fa22baf436f4815a136673b12a476c7 (DEFAULT_TABLE table=default.Table_42771479a01b45f7ade52d2ec3896c4a [id=312b160396c64566981a3b6325a8778d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:28.724601 1084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fa22baf436f4815a136673b12a476c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:28.725630 1081 tablet_service.cc:1467] Processing CreateTablet for tablet 4ab79e0a576843e89a2d524608c5b7f4 (DEFAULT_TABLE table=default.Table_4be1140a3b21462e9be7fe47af15dd09 [id=5496cdb5c870491dbf3373b3c6cd927f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:28.726505 1081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ab79e0a576843e89a2d524608c5b7f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:28.731989 465 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet 59a46bc865cc48df8e98598764fd1b54
I20250114 02:54:28.733264 465 catalog_manager.cc:6596] ReplaceTablet: tablet 59a46bc865cc48df8e98598764fd1b54 deleted and replaced by tablet cb8aa4212982417b861323c4809017d4
I20250114 02:54:28.738035 933 tablet_service.cc:1467] Processing CreateTablet for tablet 232d6a697f744e179a1a8da492a7d227 (DEFAULT_TABLE table=default.Table_403c29800daf456d93b3b60002fa6f96 [id=dcae1865f0b44cd8831a9fd0ab629415]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:28.738942 933 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 232d6a697f744e179a1a8da492a7d227. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:28.745121 940 tablet_service.cc:1514] Processing DeleteTablet for tablet 59a46bc865cc48df8e98598764fd1b54 with delete_type TABLET_DATA_DELETED (replaced by tablet cb8aa4212982417b861323c4809017d4) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:28.748021 795 tablet_service.cc:1514] Processing DeleteTablet for tablet 59a46bc865cc48df8e98598764fd1b54 with delete_type TABLET_DATA_DELETED (replaced by tablet cb8aa4212982417b861323c4809017d4) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:28.751502 1966 raft_consensus.cc:2798] T 84738a6c78b24c1fb64013048763b18f P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:28.660625 2016 raft_consensus.cc:513] T 38fe8829d98547fb907ef78099df3d7f P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:28.667416 961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "719188f639724851a5fc2b37cb60fdbb" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
W20250114 02:54:16.834726 1529 debug-util.cc:398] Leaking SignalData structure 0x7b0800006bc0 after lost signal to thread 1423
I20250114 02:54:28.771257 961 raft_consensus.cc:2463] T 719188f639724851a5fc2b37cb60fdbb P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 0.
W20250114 02:54:28.771775 1529 kernel_stack_watchdog.cc:198] Thread 1423 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:636 for 380ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250114 02:54:15.478464 1542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 02:54:28.775828 1563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 02:54:28.776137 1562 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 13319 milliseconds
W20250114 02:54:28.779318 1423 thread.cc:635] OpenStack (cloud detector) Time spent creating pthread: real 13.326s user 0.869s sys 1.662s
W20250114 02:54:28.779683 1423 thread.cc:601] OpenStack (cloud detector) Time spent starting thread: real 13.327s user 0.869s sys 1.662s
I20250114 02:54:28.779955 1423 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250114 02:54:28.781188 2016 leader_election.cc:290] T 38fe8829d98547fb907ef78099df3d7f P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Requested vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:28.781567 1423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 02:54:28.784322 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c86562688d5b4fffb71cbdf7be0afc37" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:28.784957 962 raft_consensus.cc:2463] T c86562688d5b4fffb71cbdf7be0afc37 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 0.
I20250114 02:54:28.784957 1966 raft_consensus.cc:695] T 84738a6c78b24c1fb64013048763b18f P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Becoming Leader. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Running, Role: LEADER
I20250114 02:54:28.785907 1966 consensus_queue.cc:237] T 84738a6c78b24c1fb64013048763b18f P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:28.793076 464 catalog_manager.cc:5526] T ddfbab181b1f474192c664cd0316037f P 90d6c2c58bb54e0eafb365625ac33871 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67). New cstate: current_term: 1 leader_uuid: "90d6c2c58bb54e0eafb365625ac33871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } }
I20250114 02:54:28.794610 464 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet 044832851f804208b0a371eb0857dbf3 (table default.Table_0518d8bed7304c86aff32e05e6f07c0d [id=515a2fe3f1e84e82bccc02ef5cecef1b]). Expected version 1 got 0
I20250114 02:54:28.795874 464 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet ce7d6088fc884a248918d2348adcaccf (table default.Table_428bd60f16824ac2b8bfaad6819f76e1 [id=35c5879da4b344118a377335fc6f83de]). Expected version 1 got 0
I20250114 02:54:28.802279 795 tablet_service.cc:1467] Processing CreateTablet for tablet 232d6a697f744e179a1a8da492a7d227 (DEFAULT_TABLE table=default.Table_403c29800daf456d93b3b60002fa6f96 [id=dcae1865f0b44cd8831a9fd0ab629415]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:28.803323 795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 232d6a697f744e179a1a8da492a7d227. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:28.805405 942 tablet_service.cc:1467] Processing CreateTablet for tablet 62af33f52a264fa8ba977b4a05e74d2b (DEFAULT_TABLE table=default.Table_81dd1d2ab8b741db9483ac53260431ef [id=b825f6b5a125445aa2feddab78584d73]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:28.806413 942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62af33f52a264fa8ba977b4a05e74d2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:28.815909 940 tablet_service.cc:1467] Processing CreateTablet for tablet f0e6054998df447287fdff45a6504050 (DEFAULT_TABLE table=default.Table_2850a159f211445988d540f3a56ec551 [id=441702f585ce4600be174d46b8607686]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:28.816913 940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0e6054998df447287fdff45a6504050. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:28.817378 798 tablet_service.cc:1467] Processing CreateTablet for tablet 62af33f52a264fa8ba977b4a05e74d2b (DEFAULT_TABLE table=default.Table_81dd1d2ab8b741db9483ac53260431ef [id=b825f6b5a125445aa2feddab78584d73]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:28.817875 794 tablet_service.cc:1467] Processing CreateTablet for tablet f0e6054998df447287fdff45a6504050 (DEFAULT_TABLE table=default.Table_2850a159f211445988d540f3a56ec551 [id=441702f585ce4600be174d46b8607686]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:28.818292 798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62af33f52a264fa8ba977b4a05e74d2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:28.818753 794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0e6054998df447287fdff45a6504050. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:28.820309 791 tablet_service.cc:1467] Processing CreateTablet for tablet 84edeeba97314e1abef8b13aab838e98 (DEFAULT_TABLE table=default.Table_d1de82caa5b94791b6c7dfed1c376e8a [id=03c775a54e9e443cbc4fab853bd6d49c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:28.821245 791 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84edeeba97314e1abef8b13aab838e98. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:28.825750 937 tablet_service.cc:1467] Processing CreateTablet for tablet 84edeeba97314e1abef8b13aab838e98 (DEFAULT_TABLE table=default.Table_d1de82caa5b94791b6c7dfed1c376e8a [id=03c775a54e9e443cbc4fab853bd6d49c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:28.826746 937 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84edeeba97314e1abef8b13aab838e98. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:28.842900 787 tablet_service.cc:1467] Processing CreateTablet for tablet efcb1bcd47174cde8c8ab7f99b3bf21e (DEFAULT_TABLE table=default.Table_58e014a2cfee480184ff3c66b1768cf6 [id=5745501ecd7b4cecaca3d35502a6cc40]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:28.843107 936 tablet_service.cc:1467] Processing CreateTablet for tablet efcb1bcd47174cde8c8ab7f99b3bf21e (DEFAULT_TABLE table=default.Table_58e014a2cfee480184ff3c66b1768cf6 [id=5745501ecd7b4cecaca3d35502a6cc40]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:28.844007 936 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efcb1bcd47174cde8c8ab7f99b3bf21e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:28.844015 787 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efcb1bcd47174cde8c8ab7f99b3bf21e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:28.853394 786 tablet_service.cc:1467] Processing CreateTablet for tablet 7157b8c416d34c22b75953ae1994df33 (DEFAULT_TABLE table=default.Table_bf613453528d4aad91e5ecf042f8aa21 [id=347f25d3fd324991a401377d1327c93f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:28.853516 931 tablet_service.cc:1467] Processing CreateTablet for tablet 7157b8c416d34c22b75953ae1994df33 (DEFAULT_TABLE table=default.Table_bf613453528d4aad91e5ecf042f8aa21 [id=347f25d3fd324991a401377d1327c93f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:28.854583 786 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7157b8c416d34c22b75953ae1994df33. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:28.854602 931 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7157b8c416d34c22b75953ae1994df33. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:28.858397 785 tablet_service.cc:1467] Processing CreateTablet for tablet 5ed33237db164fe0bee345090a4533e8 (DEFAULT_TABLE table=default.Table_26478578c48e45af84e6b020c897a5f2 [id=77e4cc4023c141099fa2555667dc7ed3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:28.859503 785 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ed33237db164fe0bee345090a4533e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:28.861799 784 tablet_service.cc:1467] Processing CreateTablet for tablet 270b40a2487c4b5b8bf8ee244edde376 (DEFAULT_TABLE table=default.Table_db23db370ae74af5b334c7e6b5e130c2 [id=38a5e63feaa1454abde9ee3e71414415]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:28.862951 784 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 270b40a2487c4b5b8bf8ee244edde376. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:28.869532 930 tablet_service.cc:1467] Processing CreateTablet for tablet 5ed33237db164fe0bee345090a4533e8 (DEFAULT_TABLE table=default.Table_26478578c48e45af84e6b020c897a5f2 [id=77e4cc4023c141099fa2555667dc7ed3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:28.870272 470 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet edc69b66bea54625a4b15867b75986b7 (table default.Table_0a0bda55a4a9476c98d961822b8a19bd [id=fb2be5c1ca9444a197f3315ea9c59b09]). Expected version 1 got 0
I20250114 02:54:28.870653 930 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ed33237db164fe0bee345090a4533e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:28.871618 470 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 38fe8829d98547fb907ef78099df3d7f (table default.Table_60450873174a4c80aa7e2122343d82be [id=9299683349354f879847b7c9d93f6072]). Expected version 1 got 0
I20250114 02:54:28.881757 929 tablet_service.cc:1467] Processing CreateTablet for tablet 270b40a2487c4b5b8bf8ee244edde376 (DEFAULT_TABLE table=default.Table_db23db370ae74af5b334c7e6b5e130c2 [id=38a5e63feaa1454abde9ee3e71414415]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:28.883414 929 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 270b40a2487c4b5b8bf8ee244edde376. 1 dirs total, 0 dirs full, 0 dirs failed
W20250114 02:54:28.897639 1042 leader_election.cc:343] T 84738a6c78b24c1fb64013048763b18f P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Not found: Tablet not found: 84738a6c78b24c1fb64013048763b18f
I20250114 02:54:28.898711 1042 leader_election.cc:304] T edc69b66bea54625a4b15867b75986b7 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90d6c2c58bb54e0eafb365625ac33871; no voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2
I20250114 02:54:28.898834 470 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet ce7d6088fc884a248918d2348adcaccf (table default.Table_428bd60f16824ac2b8bfaad6819f76e1 [id=35c5879da4b344118a377335fc6f83de]). Expected version 1 got 0
I20250114 02:54:28.901423 470 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet a52d6749ff834b62a8e20a2f45935240 (table default.Table_1c9b7896941e4289b1c12e20c210c2c1 [id=b3ee15b9f10f4d489edbde3af9034d25]). Expected version 1 got 0
I20250114 02:54:28.903825 2031 raft_consensus.cc:2743] T edc69b66bea54625a4b15867b75986b7 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:28.904237 470 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 92cd8eccacf34ab7a66b3bbc69df3697 (table default.Table_990b8d5f1bd146d6b358457f954780c9 [id=26627ccdc808451fb9593ad7a6df6991]). Expected version 2 got 0
I20250114 02:54:28.915161 1206 tablet_bootstrap.cc:654] T ed7f575653c848d7b076c17c38c4a244 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:28.915948 470 catalog_manager.cc:5526] T 99b940d872ce4781907517c40fe08781 P 819f9e3412e54d44abb11d8b3ca877c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65). New cstate: current_term: 1 leader_uuid: "819f9e3412e54d44abb11d8b3ca877c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:28.918912 1086 tablet_service.cc:1467] Processing CreateTablet for tablet 6cf7f6a9debe4ba3bb0b58fc5f48980c (DEFAULT_TABLE table=default.Table_8d2e7b6869da423a949c59081ea3618d [id=9cf2e181b6bc44f3b96312c76ba6e646]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:28.919073 1083 tablet_service.cc:1467] Processing CreateTablet for tablet 9da1c960cb3d4b57869a307afdb41ac3 (DEFAULT_TABLE table=default.Table_3cd1350ab3034d69b3f85470c9ab3e34 [id=ed3b05cec0a348a5badacea3e6c47ee6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:28.920058 1086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6cf7f6a9debe4ba3bb0b58fc5f48980c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:28.924813 1083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9da1c960cb3d4b57869a307afdb41ac3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:28.928607 470 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 99b940d872ce4781907517c40fe08781 (table default.Table_0943b1ca347042c9a6043006bc566701 [id=e3418787ef0a4e22979fb41d4f94ebb0]). Expected version 1 got 0
I20250114 02:54:28.929908 470 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 719188f639724851a5fc2b37cb60fdbb (table default.Table_8f8a18d7868c44bbb4d66a0692b08186 [id=e6292728826f437cb75c524a8aef2e87]). Expected version 1 got 0
I20250114 02:54:28.937294 470 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 044832851f804208b0a371eb0857dbf3 (table default.Table_0518d8bed7304c86aff32e05e6f07c0d [id=515a2fe3f1e84e82bccc02ef5cecef1b]). Expected version 1 got 0
I20250114 02:54:28.938385 466 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet 9ffec1d1bb594fda83df8d94d4533d56
I20250114 02:54:28.939719 466 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:33540 (request call id 275) took 1553 ms. Trace:
I20250114 02:54:28.940326 789 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ffec1d1bb594fda83df8d94d4533d56 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:27 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:28.942732 817 raft_consensus.cc:2463] T 6e858a8449444ef999b9bbef612249d3 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1.
I20250114 02:54:28.940080 466 rpcz_store.cc:276] 0114 02:54:27.385907 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:27.451411 (+ 65504us) service_pool.cc:225] Handling call
0114 02:54:27.452590 (+ 1179us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0114 02:54:27.453133 (+ 543us) catalog_manager.cc:2753] Modifying in-memory table state
0114 02:54:27.453273 (+ 140us) catalog_manager.cc:2762] Locking tablets
0114 02:54:27.503790 (+ 50517us) catalog_manager.cc:2776] Removing table and tablets from system table
0114 02:54:27.609926 (+106136us) catalog_manager.cc:2794] Removing table from by-name map
0114 02:54:27.610114 (+ 188us) catalog_manager.cc:2810] Committing in-memory state
0114 02:54:28.938296 (+1328182us) catalog_manager.cc:2814] Aborting table tasks
0114 02:54:28.939655 (+ 1359us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:27.552794 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:27.553139 (+ 345us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:27.553160 (+ 21us) write_op.cc:435] Acquired schema lock
0114 02:54:27.553172 (+ 12us) tablet.cc:659] Decoding operations
0114 02:54:27.553472 (+ 300us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:27.553503 (+ 31us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:27.553524 (+ 21us) tablet.cc:682] Acquiring locks for 2 operations
0114 02:54:27.553787 (+ 263us) tablet.cc:698] Row locks acquired
0114 02:54:27.553799 (+ 12us) write_op.cc:260] PREPARE: finished
0114 02:54:27.553901 (+ 102us) write_op.cc:270] Start()
0114 02:54:27.553977 (+ 76us) write_op.cc:276] Timestamp: P: 1736823267553849 usec, L: 0
0114 02:54:27.553992 (+ 15us) op_driver.cc:348] REPLICATION: starting
0114 02:54:27.554488 (+ 496us) log.cc:844] Serialized 734 byte log entry
0114 02:54:27.578167 (+ 23679us) op_driver.cc:464] REPLICATION: finished
0114 02:54:27.591540 (+ 13373us) write_op.cc:301] APPLY: starting
0114 02:54:27.591593 (+ 53us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:27.591696 (+ 103us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:27.591790 (+ 94us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:27.592029 (+ 239us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:27.592169 (+ 140us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0114 02:54:27.592246 (+ 77us) write_op.cc:312] APPLY: finished
0114 02:54:27.592457 (+ 211us) log.cc:844] Serialized 29 byte log entry
0114 02:54:27.592621 (+ 164us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:27.592700 (+ 79us) write_op.cc:454] Released schema lock
0114 02:54:27.592738 (+ 38us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":13238,"apply.run_cpu_time_us":1400,"apply.run_wall_time_us":1400,"num_ops":2,"prepare.queue_time_us":47117,"prepare.run_cpu_time_us":1947,"prepare.run_wall_time_us":3048,"raft.queue_time_us":82,"raft.run_cpu_time_us":10,"raft.run_wall_time_us":11,"replication_time_us":24102,"thread_start_us":57268,"threads_started":1}]]}
I20250114 02:54:28.948297 928 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ffec1d1bb594fda83df8d94d4533d56 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:27 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:28.904853 1992 ts_tablet_manager.cc:1905] T 1381e26725174b9c877075cf5444f566 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:28.973910 1079 tablet_service.cc:1467] Processing CreateTablet for tablet 25bce689024f49a49496e415efe91fdc (DEFAULT_TABLE table=default.Table_e18d67a6f19d4259bee6f447deced18c [id=b3d24ad0e5b7449d808b8b1826fc07c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:28.974298 1080 tablet_service.cc:1467] Processing CreateTablet for tablet 232d6a697f744e179a1a8da492a7d227 (DEFAULT_TABLE table=default.Table_403c29800daf456d93b3b60002fa6f96 [id=dcae1865f0b44cd8831a9fd0ab629415]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:28.975096 1079 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25bce689024f49a49496e415efe91fdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:28.974932 1078 tablet_service.cc:1467] Processing CreateTablet for tablet 62af33f52a264fa8ba977b4a05e74d2b (DEFAULT_TABLE table=default.Table_81dd1d2ab8b741db9483ac53260431ef [id=b825f6b5a125445aa2feddab78584d73]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:28.975288 1085 tablet_service.cc:1514] Processing DeleteTablet for tablet 59a46bc865cc48df8e98598764fd1b54 with delete_type TABLET_DATA_DELETED (replaced by tablet cb8aa4212982417b861323c4809017d4) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:28.975930 1078 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62af33f52a264fa8ba977b4a05e74d2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:28.976668 1082 tablet_service.cc:1467] Processing CreateTablet for tablet f414957450cc49fdb59776e6fd8acb5d (DEFAULT_TABLE table=default.Table_6071c0da3a0d4ecc9e063e0bab9712de [id=226c86aee94b4aa3b1569e2d4784e842]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:28.977041 1080 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 232d6a697f744e179a1a8da492a7d227. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:28.977551 1082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f414957450cc49fdb59776e6fd8acb5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:29.000142 465 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet edc69b66bea54625a4b15867b75986b7 (table default.Table_0a0bda55a4a9476c98d961822b8a19bd [id=fb2be5c1ca9444a197f3315ea9c59b09]). Expected version 1 got 0
I20250114 02:54:29.005240 465 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 719188f639724851a5fc2b37cb60fdbb (table default.Table_8f8a18d7868c44bbb4d66a0692b08186 [id=e6292728826f437cb75c524a8aef2e87]). Expected version 1 got 0
I20250114 02:54:29.006001 465 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet ce7d6088fc884a248918d2348adcaccf (table default.Table_428bd60f16824ac2b8bfaad6819f76e1 [id=35c5879da4b344118a377335fc6f83de]). Expected version 1 got 0
I20250114 02:54:29.008621 465 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 044832851f804208b0a371eb0857dbf3 (table default.Table_0518d8bed7304c86aff32e05e6f07c0d [id=515a2fe3f1e84e82bccc02ef5cecef1b]). Expected version 1 got 0
I20250114 02:54:29.009163 465 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 38fe8829d98547fb907ef78099df3d7f (table default.Table_60450873174a4c80aa7e2122343d82be [id=9299683349354f879847b7c9d93f6072]). Expected version 1 got 0
I20250114 02:54:29.010013 465 catalog_manager.cc:5526] T 92cd8eccacf34ab7a66b3bbc69df3697 P 525198593c8e47afaee54e74267b87e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 525198593c8e47afaee54e74267b87e3 (127.0.107.66). New cstate: current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:29.011245 465 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 92cd8eccacf34ab7a66b3bbc69df3697 (table default.Table_990b8d5f1bd146d6b358457f954780c9 [id=26627ccdc808451fb9593ad7a6df6991]). Expected version 2 got 0
I20250114 02:54:29.011946 465 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 99b940d872ce4781907517c40fe08781 (table default.Table_0943b1ca347042c9a6043006bc566701 [id=e3418787ef0a4e22979fb41d4f94ebb0]). Expected version 1 got 0
I20250114 02:54:29.016924 1077 tablet_service.cc:1467] Processing CreateTablet for tablet 7157b8c416d34c22b75953ae1994df33 (DEFAULT_TABLE table=default.Table_bf613453528d4aad91e5ecf042f8aa21 [id=347f25d3fd324991a401377d1327c93f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:29.017151 1085 tablet_service.cc:1467] Processing CreateTablet for tablet efcb1bcd47174cde8c8ab7f99b3bf21e (DEFAULT_TABLE table=default.Table_58e014a2cfee480184ff3c66b1768cf6 [id=5745501ecd7b4cecaca3d35502a6cc40]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:29.018008 1077 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7157b8c416d34c22b75953ae1994df33. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:29.019553 1084 tablet_service.cc:1467] Processing CreateTablet for tablet f0e6054998df447287fdff45a6504050 (DEFAULT_TABLE table=default.Table_2850a159f211445988d540f3a56ec551 [id=441702f585ce4600be174d46b8607686]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:29.022261 1075 tablet_service.cc:1467] Processing CreateTablet for tablet 270b40a2487c4b5b8bf8ee244edde376 (DEFAULT_TABLE table=default.Table_db23db370ae74af5b334c7e6b5e130c2 [id=38a5e63feaa1454abde9ee3e71414415]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:29.023263 1074 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ffec1d1bb594fda83df8d94d4533d56 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:27 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:29.024555 1076 tablet_service.cc:1467] Processing CreateTablet for tablet 5ed33237db164fe0bee345090a4533e8 (DEFAULT_TABLE table=default.Table_26478578c48e45af84e6b020c897a5f2 [id=77e4cc4023c141099fa2555667dc7ed3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:29.031502 1080 tablet_service.cc:1467] Processing CreateTablet for tablet 84edeeba97314e1abef8b13aab838e98 (DEFAULT_TABLE table=default.Table_d1de82caa5b94791b6c7dfed1c376e8a [id=03c775a54e9e443cbc4fab853bd6d49c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:29.035212 2016 raft_consensus.cc:491] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:29.035643 2016 raft_consensus.cc:513] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
W20250114 02:54:29.037442 1207 thread.cc:635] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.623s user 0.021s sys 0.041s
I20250114 02:54:29.037591 2016 leader_election.cc:290] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
W20250114 02:54:29.037695 1207 thread.cc:601] log-alloc [worker] (thread pool) Time spent starting thread: real 0.623s user 0.021s sys 0.041s
I20250114 02:54:29.039459 1207 tablet_bootstrap.cc:492] T d05089eae2b74f9fb23a226d358a59da P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:29.039819 1207 ts_tablet_manager.cc:1397] T d05089eae2b74f9fb23a226d358a59da P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.650s user 0.027s sys 0.042s
W20250114 02:54:29.048910 898 leader_election.cc:343] T 38fe8829d98547fb907ef78099df3d7f P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:29.054610 1075 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 270b40a2487c4b5b8bf8ee244edde376. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:29.056113 1086 tablet_service.cc:1467] Processing CreateTablet for tablet d8d16aad0fed4f44b4a479a3e4f77bc9 (DEFAULT_TABLE table=default.Table_26355cc977e54afca705ed400dae8207 [id=0436dc9f10d74e7eb434ac41f704af39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:29.057014 1086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8d16aad0fed4f44b4a479a3e4f77bc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:29.057861 1423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 02:54:29.057961 1085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efcb1bcd47174cde8c8ab7f99b3bf21e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:29.061394 1423 hybrid_clock.cc:648] HybridClock initialized: now 1736823269061319 us; error 91 us; skew 500 ppm
I20250114 02:54:29.062681 1423 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
W20250114 02:54:29.066880 1042 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.65:39397, user_credentials={real_user=slave}} blocked reactor thread for 168264us
I20250114 02:54:29.068688 1080 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84edeeba97314e1abef8b13aab838e98. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:29.069178 1042 leader_election.cc:304] T 6e858a8449444ef999b9bbef612249d3 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters: 525198593c8e47afaee54e74267b87e3
I20250114 02:54:29.069912 1966 raft_consensus.cc:491] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:29.070374 1966 raft_consensus.cc:513] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:29.030512 1990 ts_tablet_manager.cc:1918] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:29.075966 1990 log.cc:1198] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/f0bd9bd5a17d418d8082ed0ebd1bf4cc
I20250114 02:54:29.076835 1990 ts_tablet_manager.cc:1939] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
I20250114 02:54:29.077010 2121 raft_consensus.cc:2798] T 6e858a8449444ef999b9bbef612249d3 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:29.077458 2121 raft_consensus.cc:695] T 6e858a8449444ef999b9bbef612249d3 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Becoming Leader. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Running, Role: LEADER
I20250114 02:54:29.078215 2121 consensus_queue.cc:237] T 6e858a8449444ef999b9bbef612249d3 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:29.083101 1076 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ed33237db164fe0bee345090a4533e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:29.050392 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59a46bc865cc48df8e98598764fd1b54" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:29.096717 1084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0e6054998df447287fdff45a6504050. 1 dirs total, 0 dirs full, 0 dirs failed
W20250114 02:54:29.097579 898 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.67:45749, user_credentials={real_user=slave}} blocked reactor thread for 48689.5us
I20250114 02:54:29.109355 1990 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 251) took 3494 ms. Trace:
I20250114 02:54:29.109826 1990 rpcz_store.cc:276] 0114 02:54:25.614550 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:25.616309 (+ 1759us) service_pool.cc:225] Handling call
0114 02:54:27.817865 (+2201556us) ts_tablet_manager.cc:1145] Deleting tablet f0bd9bd5a17d418d8082ed0ebd1bf4cc
0114 02:54:27.817913 (+ 48us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:28.343510 (+525597us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:29.030321 (+686811us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:29.109296 (+ 78975us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2171513}
W20250114 02:54:29.112828 1205 thread.cc:635] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.526s user 0.000s sys 0.017s
W20250114 02:54:29.113096 1205 thread.cc:601] log-alloc [worker] (thread pool) Time spent starting thread: real 0.526s user 0.000s sys 0.017s
I20250114 02:54:29.072232 1966 leader_election.cc:290] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:29.114113 2016 consensus_queue.cc:1035] T 92cd8eccacf34ab7a66b3bbc69df3697 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:29.114611 1207 raft_consensus.cc:357] T d05089eae2b74f9fb23a226d358a59da P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:29.115267 1207 raft_consensus.cc:383] T d05089eae2b74f9fb23a226d358a59da P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:29.115512 1207 raft_consensus.cc:738] T d05089eae2b74f9fb23a226d358a59da P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:29.123109 814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32dd43e8a3544dbaa6e2fbb4b4c1dc28" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:29.123739 814 raft_consensus.cc:2463] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 0.
I20250114 02:54:29.142711 1042 leader_election.cc:304] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:29.147042 1205 tablet_bootstrap.cc:492] T df045d80d8ce4ff194a6d2039a44bce6 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:29.147470 1205 ts_tablet_manager.cc:1397] T df045d80d8ce4ff194a6d2039a44bce6 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.570s user 0.000s sys 0.027s
I20250114 02:54:29.151180 1820 raft_consensus.cc:2798] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:29.151539 1820 raft_consensus.cc:491] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:29.151840 1820 raft_consensus.cc:3054] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:29.162587 1205 raft_consensus.cc:357] T df045d80d8ce4ff194a6d2039a44bce6 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:29.163261 1205 raft_consensus.cc:383] T df045d80d8ce4ff194a6d2039a44bce6 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:29.163520 1205 raft_consensus.cc:738] T df045d80d8ce4ff194a6d2039a44bce6 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:29.164047 1205 consensus_queue.cc:260] T df045d80d8ce4ff194a6d2039a44bce6 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:29.168948 1820 raft_consensus.cc:513] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:29.171896 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32dd43e8a3544dbaa6e2fbb4b4c1dc28" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:29.172493 962 raft_consensus.cc:2463] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 0.
I20250114 02:54:29.172713 466 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet 92cd8eccacf34ab7a66b3bbc69df3697 (table default.Table_990b8d5f1bd146d6b358457f954780c9 [id=26627ccdc808451fb9593ad7a6df6991]). Expected version 2 got 0
I20250114 02:54:29.173683 466 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet 044832851f804208b0a371eb0857dbf3 (table default.Table_0518d8bed7304c86aff32e05e6f07c0d [id=515a2fe3f1e84e82bccc02ef5cecef1b]). Expected version 1 got 0
I20250114 02:54:29.179173 466 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet ce7d6088fc884a248918d2348adcaccf (table default.Table_428bd60f16824ac2b8bfaad6819f76e1 [id=35c5879da4b344118a377335fc6f83de]). Expected version 1 got 0
I20250114 02:54:29.184862 467 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:33540 (request call id 266) took 2431 ms. Trace:
I20250114 02:54:29.185165 467 rpcz_store.cc:276] 0114 02:54:26.753016 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:26.759450 (+ 6434us) service_pool.cc:225] Handling call
0114 02:54:26.843824 (+ 84374us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0114 02:54:26.844770 (+ 946us) catalog_manager.cc:3627] Apply alter schema
0114 02:54:26.845415 (+ 645us) catalog_manager.cc:3659] Acquired catalog manager lock
0114 02:54:26.845671 (+ 256us) catalog_manager.cc:3817] Updating metadata on disk
0114 02:54:26.889741 (+ 44070us) spinlock_profiling.cc:243] Waited 41.9 ms on lock 0x7b5400031480. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f6461ae1a81 00007f6461bb59bc 00007f6461bab5dd 00007f6461b6cc5b 00007f6461b6ca34 00007f646351d3a1 00007f646351dbc6 00007f64633dd364 00007f64633d360d 00007f64634ec9c2 00007f6461259644
0114 02:54:27.230452 (+340711us) catalog_manager.cc:3848] Committing alterations to in-memory state
0114 02:54:29.184801 (+1954349us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:26.930641 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:26.931001 (+ 360us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:26.931023 (+ 22us) write_op.cc:435] Acquired schema lock
0114 02:54:26.931037 (+ 14us) tablet.cc:659] Decoding operations
0114 02:54:26.931259 (+ 222us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:26.931290 (+ 31us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:26.931309 (+ 19us) tablet.cc:682] Acquiring locks for 1 operations
0114 02:54:26.931451 (+ 142us) tablet.cc:698] Row locks acquired
0114 02:54:26.931463 (+ 12us) write_op.cc:260] PREPARE: finished
0114 02:54:26.931560 (+ 97us) write_op.cc:270] Start()
0114 02:54:26.931635 (+ 75us) write_op.cc:276] Timestamp: P: 1736823266931526 usec, L: 0
0114 02:54:26.931649 (+ 14us) op_driver.cc:348] REPLICATION: starting
0114 02:54:26.937180 (+ 5531us) log.cc:844] Serialized 405 byte log entry
0114 02:54:27.220877 (+283697us) op_driver.cc:464] REPLICATION: finished
0114 02:54:27.221491 (+ 614us) write_op.cc:301] APPLY: starting
0114 02:54:27.221540 (+ 49us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:27.221626 (+ 86us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:27.221705 (+ 79us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:27.221862 (+ 157us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:27.221982 (+ 120us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:27.222025 (+ 43us) write_op.cc:312] APPLY: finished
0114 02:54:27.222217 (+ 192us) log.cc:844] Serialized 23 byte log entry
0114 02:54:27.222351 (+ 134us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:27.222415 (+ 64us) write_op.cc:454] Released schema lock
0114 02:54:27.222452 (+ 37us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:27.222683 (+ 231us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:27.222719 (+ 36us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":96409344,"child_traces":[["op",{"apply.queue_time_us":488,"apply.run_cpu_time_us":1153,"apply.run_wall_time_us":1152,"num_ops":1,"prepare.queue_time_us":40708,"prepare.run_cpu_time_us":1675,"prepare.run_wall_time_us":7930,"replication_time_us":289141}]]}
I20250114 02:54:29.200582 817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32dd43e8a3544dbaa6e2fbb4b4c1dc28" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:29.201205 817 raft_consensus.cc:3054] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:29.206652 778 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:53068 (request call id 31) took 1030 ms. Trace:
I20250114 02:54:29.206991 778 rpcz_store.cc:276] 0114 02:54:28.176160 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:28.180410 (+ 4250us) service_pool.cc:225] Handling call
0114 02:54:28.371849 (+191439us) spinlock_profiling.cc:243] Waited 161 ms on lock 0x7b54000b5754. stack: 00007f66faa37f63 00007f66faa37e60 00007f66fa5ae584 00000000004fcd24 00000000004fccb9 00007f67019386d1 00007f67019ad268 00007f67029a36bd 00007f6701241e14 00007f6701241da2 00007f6701241cd1 00007f6701241c4c 00007f6701240f02 00007f66ff780dfc 00007f66ff780236 00007f66ff77fbdf
0114 02:54:28.990068 (+618219us) spinlock_profiling.cc:243] Waited 182 ms on lock 0x7b54000f24a8. stack: 00007f66faa37f63 00007f66faa37e60 00007f66fa5ae584 00000000004fcd24 00000000004fccb9 00007f6701d4fa81 00007f6701ddb151 00007f67029a3690 00007f6701241e14 00007f6701241da2 00007f6701241cd1 00007f6701241c4c 00007f6701240f02 00007f66ff780dfc 00007f66ff780236 00007f66ff77fbdf
0114 02:54:29.206464 (+216396us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":790262912}
W20250114 02:54:29.220547 898 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.67:45749, user_credentials={real_user=slave}} blocked reactor thread for 101704us
W20250114 02:54:29.221172 898 leader_election.cc:343] T 719188f639724851a5fc2b37cb60fdbb P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:29.221722 898 leader_election.cc:304] T edc69b66bea54625a4b15867b75986b7 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:29.222498 2016 raft_consensus.cc:2798] T edc69b66bea54625a4b15867b75986b7 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:29.223068 2016 raft_consensus.cc:695] T edc69b66bea54625a4b15867b75986b7 P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Becoming Leader. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Running, Role: LEADER
I20250114 02:54:29.223861 2016 consensus_queue.cc:237] T edc69b66bea54625a4b15867b75986b7 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:29.224977 2140 tablet.cc:1721] T 044832851f804208b0a371eb0857dbf3 P 525198593c8e47afaee54e74267b87e3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250114 02:54:29.227768 1990 tablet_replica.cc:331] T 46360f726fad4534a2daf67581346fc1 P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:29.227768 811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7774c2087422492696a58e0075921ae7" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:29.228374 811 raft_consensus.cc:3054] T 7774c2087422492696a58e0075921ae7 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:29.228744 1990 raft_consensus.cc:2238] T 46360f726fad4534a2daf67581346fc1 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Raft consensus shutting down.
I20250114 02:54:29.182006 1820 leader_election.cc:290] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Requested vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:29.231228 1103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7774c2087422492696a58e0075921ae7" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:29.232550 1104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c86562688d5b4fffb71cbdf7be0afc37" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:29.232051 1103 raft_consensus.cc:3054] T 7774c2087422492696a58e0075921ae7 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:29.238569 813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c86562688d5b4fffb71cbdf7be0afc37" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:29.239159 813 raft_consensus.cc:2463] T c86562688d5b4fffb71cbdf7be0afc37 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:29.239331 465 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: default.Table_e4cc8164591444a4bf1e683f3a609d36 [id=22b7edcf4b7e4c31b3c23dc44ccf028e] alter complete (version 2)
I20250114 02:54:29.241923 1103 raft_consensus.cc:2463] T 7774c2087422492696a58e0075921ae7 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
W20250114 02:54:29.248548 455 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 6e858a8449444ef999b9bbef612249d3 (table default.Table_5a9480c6474848d6b400e7dd274d15b0 [id=a17ad30ef32f4cbcb22b5a7d1c7096ff], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 6e858a8449444ef999b9bbef612249d3 consensus state has no leader
I20250114 02:54:29.256523 811 raft_consensus.cc:2463] T 7774c2087422492696a58e0075921ae7 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:29.256825 469 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: default.Table_0931c459ae794d2091fcc0f51d031548 [id=e279c31bd63b40b9ade03a3b928bf153] alter complete (version 2)
I20250114 02:54:29.259312 1206 tablet_bootstrap.cc:492] T ed7f575653c848d7b076c17c38c4a244 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:29.263195 1104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38fe8829d98547fb907ef78099df3d7f" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:29.260144 751 leader_election.cc:304] T 719188f639724851a5fc2b37cb60fdbb P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters: 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:29.264910 1206 ts_tablet_manager.cc:1397] T ed7f575653c848d7b076c17c38c4a244 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.802s user 0.012s sys 0.003s
I20250114 02:54:29.265236 2093 raft_consensus.cc:2798] T 719188f639724851a5fc2b37cb60fdbb P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:29.265576 2093 raft_consensus.cc:491] T 719188f639724851a5fc2b37cb60fdbb P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:29.265858 2093 raft_consensus.cc:3054] T 719188f639724851a5fc2b37cb60fdbb P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:29.266348 817 raft_consensus.cc:2463] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1.
I20250114 02:54:29.267125 1206 raft_consensus.cc:357] T ed7f575653c848d7b076c17c38c4a244 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:29.267781 1206 raft_consensus.cc:383] T ed7f575653c848d7b076c17c38c4a244 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:29.180032 1207 consensus_queue.cc:260] T d05089eae2b74f9fb23a226d358a59da P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:29.269260 751 leader_election.cc:304] T c86562688d5b4fffb71cbdf7be0afc37 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters: 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:29.269922 1207 ts_tablet_manager.cc:1428] T d05089eae2b74f9fb23a226d358a59da P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.230s user 0.009s sys 0.000s
I20250114 02:54:29.269958 2090 raft_consensus.cc:2798] T c86562688d5b4fffb71cbdf7be0afc37 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:29.270239 2090 raft_consensus.cc:491] T c86562688d5b4fffb71cbdf7be0afc37 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:29.270534 2090 raft_consensus.cc:3054] T c86562688d5b4fffb71cbdf7be0afc37 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:29.270757 1207 tablet_bootstrap.cc:492] T 38fe8829d98547fb907ef78099df3d7f P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:29.272133 811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32dd43e8a3544dbaa6e2fbb4b4c1dc28" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
W20250114 02:54:29.272645 463 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.67:34825 (request call id 48) took 3659 ms (client timeout 1000 ms). Trace:
I20250114 02:54:29.272841 811 raft_consensus.cc:2388] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 525198593c8e47afaee54e74267b87e3 in current term 1: Already voted for candidate 90d6c2c58bb54e0eafb365625ac33871 in this term.
I20250114 02:54:29.260998 815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38fe8829d98547fb907ef78099df3d7f" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:29.273578 815 raft_consensus.cc:3054] T 38fe8829d98547fb907ef78099df3d7f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:29.274343 1965 ts_tablet_manager.cc:1918] T de4ca83a69704a368a25954cdc1b9392 P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
W20250114 02:54:29.274600 751 consensus_peers.cc:487] T d3fb28717c6f4844af84bb817fe7dfc4 P 819f9e3412e54d44abb11d8b3ca877c2 -> Peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Couldn't send request to peer 525198593c8e47afaee54e74267b87e3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d3fb28717c6f4844af84bb817fe7dfc4. This is attempt 1: this message will repeat every 5th retry.
W20250114 02:54:29.272982 463 rpcz_store.cc:269] 0114 02:54:25.613325 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:25.622012 (+ 8687us) service_pool.cc:225] Handling call
0114 02:54:25.775210 (+153198us) spinlock_profiling.cc:243] Waited 82.8 ms on lock 0x7b54000316d4. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616cad50 00007f646174df2a 00007f64633f2dec 00007f64634eadee 00007f6461248fc4 00007f6461248f52 00007f6461248e81 00007f6461248dfc 00007f64612480b2 00007f645f512dfc 00007f645f512236
0114 02:54:26.572991 (+797781us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:26.573015 (+ 24us) write_op.cc:454] Released schema lock
0114 02:54:29.272572 (+2699557us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:26.557162 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:26.557497 (+ 335us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:26.557520 (+ 23us) write_op.cc:435] Acquired schema lock
0114 02:54:26.557532 (+ 12us) tablet.cc:659] Decoding operations
0114 02:54:26.557807 (+ 275us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:26.557847 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:26.557871 (+ 24us) tablet.cc:682] Acquiring locks for 1 operations
0114 02:54:26.558003 (+ 132us) tablet.cc:698] Row locks acquired
0114 02:54:26.558016 (+ 13us) write_op.cc:260] PREPARE: finished
0114 02:54:26.558120 (+ 104us) write_op.cc:270] Start()
0114 02:54:26.558237 (+ 117us) write_op.cc:276] Timestamp: P: 1736823266558067 usec, L: 0
0114 02:54:26.558258 (+ 21us) op_driver.cc:348] REPLICATION: starting
0114 02:54:26.558781 (+ 523us) log.cc:844] Serialized 530 byte log entry
0114 02:54:26.565489 (+ 6708us) op_driver.cc:464] REPLICATION: finished
0114 02:54:26.566447 (+ 958us) write_op.cc:301] APPLY: starting
0114 02:54:26.566513 (+ 66us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:26.566599 (+ 86us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:26.566698 (+ 99us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:26.566917 (+ 219us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:26.567085 (+ 168us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:26.567153 (+ 68us) write_op.cc:312] APPLY: finished
0114 02:54:26.567363 (+ 210us) log.cc:844] Serialized 23 byte log entry
0114 02:54:26.567500 (+ 137us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:26.567557 (+ 57us) write_op.cc:454] Released schema lock
0114 02:54:26.567584 (+ 27us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":79945,"reported_tablets":31,"spinlock_wait_cycles":190507904,"child_traces":[["op",{"apply.queue_time_us":797,"apply.run_cpu_time_us":1291,"apply.run_wall_time_us":1291,"num_ops":1,"prepare.queue_time_us":98,"prepare.run_cpu_time_us":1927,"prepare.run_wall_time_us":1927,"raft.queue_time_us":456,"raft.run_cpu_time_us":586,"raft.run_wall_time_us":585,"replication_time_us":7122}]]}
I20250114 02:54:29.280530 1965 log.cc:1198] T de4ca83a69704a368a25954cdc1b9392 P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/de4ca83a69704a368a25954cdc1b9392
I20250114 02:54:29.282959 1965 ts_tablet_manager.cc:1939] T de4ca83a69704a368a25954cdc1b9392 P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:29.291249 815 raft_consensus.cc:2463] T 38fe8829d98547fb907ef78099df3d7f P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:29.300763 1990 raft_consensus.cc:2267] T 46360f726fad4534a2daf67581346fc1 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:29.303570 1990 ts_tablet_manager.cc:1905] T 46360f726fad4534a2daf67581346fc1 P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:29.307222 2143 consensus_queue.cc:1035] T e15e6e52d7294aeeb4463b3a76022ec3 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:29.309458 898 leader_election.cc:304] T 719188f639724851a5fc2b37cb60fdbb P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters: 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:29.310190 2141 raft_consensus.cc:2798] T 719188f639724851a5fc2b37cb60fdbb P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:29.310578 2141 raft_consensus.cc:491] T 719188f639724851a5fc2b37cb60fdbb P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:29.310858 2141 raft_consensus.cc:3054] T 719188f639724851a5fc2b37cb60fdbb P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:29.317639 2141 raft_consensus.cc:513] T 719188f639724851a5fc2b37cb60fdbb P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:29.319880 2141 leader_election.cc:290] T 719188f639724851a5fc2b37cb60fdbb P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Requested vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:29.321218 1965 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55068 (request call id 267) took 2787 ms. Trace:
I20250114 02:54:29.326530 1103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32dd43e8a3544dbaa6e2fbb4b4c1dc28" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:29.327368 1103 raft_consensus.cc:2388] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 525198593c8e47afaee54e74267b87e3 in current term 1: Already voted for candidate 90d6c2c58bb54e0eafb365625ac33871 in this term.
I20250114 02:54:29.328198 464 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: default.Table_4c3843233dda407395ab9652d5870bc6 [id=0d173fdb502f482fb86b6d3e5fdc66f1] alter complete (version 1)
I20250114 02:54:29.292042 2093 raft_consensus.cc:513] T 719188f639724851a5fc2b37cb60fdbb P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:29.294340 2090 raft_consensus.cc:513] T c86562688d5b4fffb71cbdf7be0afc37 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:29.297286 2152 consensus_queue.cc:1035] T 99b940d872ce4781907517c40fe08781 P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:54:29.268014 1206 raft_consensus.cc:738] T ed7f575653c848d7b076c17c38c4a244 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:29.334754 1042 leader_election.cc:304] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:29.335227 1206 consensus_queue.cc:260] T ed7f575653c848d7b076c17c38c4a244 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:29.336361 2133 raft_consensus.cc:2798] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:29.336853 2133 raft_consensus.cc:695] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Becoming Leader. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Running, Role: LEADER
I20250114 02:54:29.337810 2133 consensus_queue.cc:237] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:29.322890 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32dd43e8a3544dbaa6e2fbb4b4c1dc28" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
I20250114 02:54:29.340641 962 raft_consensus.cc:3054] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:29.343333 935 tablet_service.cc:1514] Processing DeleteTablet for tablet ca986f81ed6848a89588366f3f400eec with delete_type TABLET_DATA_DELETED (replaced by tablet a2d68eccaaa64511a9ae36f2b07a7b59) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:29.346403 935 tablet_service.cc:1514] Processing DeleteTablet for tablet 1381e26725174b9c877075cf5444f566 with delete_type TABLET_DATA_DELETED (replaced by tablet 9ffec1d1bb594fda83df8d94d4533d56) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:29.347509 962 raft_consensus.cc:2463] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1.
I20250114 02:54:29.305835 1205 ts_tablet_manager.cc:1428] T df045d80d8ce4ff194a6d2039a44bce6 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.158s user 0.000s sys 0.006s
I20250114 02:54:29.348290 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet f0bd9bd5a17d418d8082ed0ebd1bf4cc (table default.Table_dc087fb027f74c4a8bd5dede1142d97d [id=971cb3ca793e4945ad07e6be966e01f3]) successfully deleted
I20250114 02:54:29.339587 934 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ac7708550954a209e2a8d217c3b39bd with delete_type TABLET_DATA_DELETED (replaced by tablet d0cc8b9e3aee4292bc69e8053d7e8cc1) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:29.349031 1205 tablet_bootstrap.cc:492] T a52d6749ff834b62a8e20a2f45935240 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:29.349054 935 tablet_service.cc:1514] Processing DeleteTablet for tablet 84738a6c78b24c1fb64013048763b18f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:23 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:29.352300 1965 rpcz_store.cc:276] 0114 02:54:26.533520 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:26.533667 (+ 147us) service_pool.cc:225] Handling call
0114 02:54:28.326101 (+1792434us) ts_tablet_manager.cc:1145] Deleting tablet de4ca83a69704a368a25954cdc1b9392
0114 02:54:28.326144 (+ 43us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:29.044722 (+718578us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:29.155626 (+110904us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:29.321145 (+165519us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2633,"tablet-delete.queue_time_us":1777349}
I20250114 02:54:29.365099 1081 tablet_service.cc:1514] Processing DeleteTablet for tablet bc46815b4803418c95ea9c039a4fae88 with delete_type TABLET_DATA_DELETED (replaced by tablet 97f4e061d3b94ee9982410b7ce135453) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:29.368809 1081 tablet_service.cc:1514] Processing DeleteTablet for tablet ca986f81ed6848a89588366f3f400eec with delete_type TABLET_DATA_DELETED (replaced by tablet a2d68eccaaa64511a9ae36f2b07a7b59) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:29.374397 1076 tablet_service.cc:1514] Processing DeleteTablet for tablet 321b9ad6d1ac4d1c80c7a01faf2015af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:23 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:29.375764 1076 tablet_service.cc:1514] Processing DeleteTablet for tablet 266da813b4514329be0ad97d07b7114f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:19 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:29.360532 898 leader_election.cc:304] T c86562688d5b4fffb71cbdf7be0afc37 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
I20250114 02:54:29.381049 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet de4ca83a69704a368a25954cdc1b9392 (table default.Table_69b77a6fd8ad45cc9e3dc32b202d93cd [id=8536eb7f6bf542daa88a8db0e4873456]) successfully deleted
I20250114 02:54:29.381507 1965 tablet_replica.cc:331] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:29.381943 898 leader_election.cc:304] T 7774c2087422492696a58e0075921ae7 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
I20250114 02:54:29.382284 1965 raft_consensus.cc:2238] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:29.382614 2097 raft_consensus.cc:2798] T c86562688d5b4fffb71cbdf7be0afc37 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:29.382771 1965 raft_consensus.cc:2267] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:29.382933 2097 raft_consensus.cc:491] T c86562688d5b4fffb71cbdf7be0afc37 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:29.383137 2142 raft_consensus.cc:2798] T 7774c2087422492696a58e0075921ae7 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:29.383257 2097 raft_consensus.cc:3054] T c86562688d5b4fffb71cbdf7be0afc37 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:29.383538 2142 raft_consensus.cc:695] T 7774c2087422492696a58e0075921ae7 P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Becoming Leader. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Running, Role: LEADER
I20250114 02:54:29.384299 2142 consensus_queue.cc:237] T 7774c2087422492696a58e0075921ae7 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:29.387284 934 tablet_service.cc:1514] Processing DeleteTablet for tablet d3fb28717c6f4844af84bb817fe7dfc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:25 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:29.392870 2097 raft_consensus.cc:513] T c86562688d5b4fffb71cbdf7be0afc37 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:29.394770 2097 leader_election.cc:290] T c86562688d5b4fffb71cbdf7be0afc37 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Requested vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:29.395154 1207 tablet_bootstrap.cc:654] T 38fe8829d98547fb907ef78099df3d7f P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
W20250114 02:54:29.399672 468 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.65:49805 (request call id 52) took 2789 ms (client timeout 1000 ms). Trace:
I20250114 02:54:29.403313 1820 raft_consensus.cc:491] T d05089eae2b74f9fb23a226d358a59da P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:29.403594 2093 leader_election.cc:290] T 719188f639724851a5fc2b37cb60fdbb P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Requested vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:29.403800 1820 raft_consensus.cc:513] T d05089eae2b74f9fb23a226d358a59da P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:29.405956 1820 leader_election.cc:290] T d05089eae2b74f9fb23a226d358a59da P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:29.364449 935 tablet_service.cc:1514] Processing DeleteTablet for tablet 46360f726fad4534a2daf67581346fc1 with delete_type TABLET_DATA_DELETED (replaced by tablet be7aad41ccdd4ef9bcfe8a985155708f) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:29.361634 2016 consensus_queue.cc:1035] T 92cd8eccacf34ab7a66b3bbc69df3697 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:54:29.407902 2090 leader_election.cc:290] T c86562688d5b4fffb71cbdf7be0afc37 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Requested vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
W20250114 02:54:29.407925 468 rpcz_store.cc:269] 0114 02:54:26.609950 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:26.613900 (+ 3950us) service_pool.cc:225] Handling call
0114 02:54:29.399591 (+2785691us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":126139,"reported_tablets":27}
I20250114 02:54:29.408460 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d05089eae2b74f9fb23a226d358a59da" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
W20250114 02:54:29.409938 1040 leader_election.cc:343] T d05089eae2b74f9fb23a226d358a59da P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
W20250114 02:54:29.406315 469 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.65:49805 (request call id 51) took 3821 ms (client timeout 1000 ms). Trace:
I20250114 02:54:29.414868 1990 ts_tablet_manager.cc:1918] T 46360f726fad4534a2daf67581346fc1 P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:29.414902 1992 ts_tablet_manager.cc:1918] T 1381e26725174b9c877075cf5444f566 P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250114 02:54:29.415285 1992 log.cc:1198] T 1381e26725174b9c877075cf5444f566 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/1381e26725174b9c877075cf5444f566
W20250114 02:54:29.410521 469 rpcz_store.cc:269] 0114 02:54:25.584836 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:25.614873 (+ 30037us) service_pool.cc:225] Handling call
0114 02:54:25.652552 (+ 37679us) spinlock_profiling.cc:243] Waited 6.49 ms on lock 0x7b54000316d4. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616cad50 00007f646174df2a 00007f64633f2dec 00007f64634eadee 00007f6461248fc4 00007f6461248f52 00007f6461248e81 00007f6461248dfc 00007f64612480b2 00007f645f512dfc 00007f645f512236
0114 02:54:26.449028 (+796476us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:26.449088 (+ 60us) write_op.cc:454] Released schema lock
0114 02:54:29.406230 (+2957142us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:26.417672 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:26.417999 (+ 327us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:26.418020 (+ 21us) write_op.cc:435] Acquired schema lock
0114 02:54:26.418031 (+ 11us) tablet.cc:659] Decoding operations
0114 02:54:26.418260 (+ 229us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:26.418296 (+ 36us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:26.418314 (+ 18us) tablet.cc:682] Acquiring locks for 1 operations
0114 02:54:26.418441 (+ 127us) tablet.cc:698] Row locks acquired
0114 02:54:26.418455 (+ 14us) write_op.cc:260] PREPARE: finished
0114 02:54:26.418558 (+ 103us) write_op.cc:270] Start()
0114 02:54:26.418643 (+ 85us) write_op.cc:276] Timestamp: P: 1736823266418519 usec, L: 0
0114 02:54:26.418659 (+ 16us) op_driver.cc:348] REPLICATION: starting
0114 02:54:26.419092 (+ 433us) log.cc:844] Serialized 356 byte log entry
0114 02:54:26.425750 (+ 6658us) op_driver.cc:464] REPLICATION: finished
0114 02:54:26.431445 (+ 5695us) write_op.cc:301] APPLY: starting
0114 02:54:26.431498 (+ 53us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:26.431591 (+ 93us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:26.431689 (+ 98us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:26.431872 (+ 183us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:26.432005 (+ 133us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:26.432048 (+ 43us) write_op.cc:312] APPLY: finished
0114 02:54:26.432249 (+ 201us) log.cc:844] Serialized 23 byte log entry
0114 02:54:26.432387 (+ 138us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:26.432534 (+ 147us) write_op.cc:454] Released schema lock
0114 02:54:26.432569 (+ 35us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0114 02:54:26.764428 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:26.766454 (+ 2026us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:26.766577 (+ 123us) write_op.cc:435] Acquired schema lock
0114 02:54:26.766589 (+ 12us) tablet.cc:659] Decoding operations
0114 02:54:26.768325 (+ 1736us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:26.768776 (+ 451us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:26.769007 (+ 231us) tablet.cc:682] Acquiring locks for 1 operations
0114 02:54:26.770982 (+ 1975us) tablet.cc:698] Row locks acquired
0114 02:54:26.771218 (+ 236us) write_op.cc:260] PREPARE: finished
0114 02:54:26.772343 (+ 1125us) write_op.cc:270] Start()
0114 02:54:26.772945 (+ 602us) write_op.cc:276] Timestamp: P: 1736823266772065 usec, L: 0
0114 02:54:26.773121 (+ 176us) op_driver.cc:348] REPLICATION: starting
0114 02:54:26.780430 (+ 7309us) log.cc:844] Serialized 356 byte log entry
0114 02:54:26.794477 (+ 14047us) op_driver.cc:464] REPLICATION: finished
0114 02:54:26.794707 (+ 230us) write_op.cc:301] APPLY: starting
0114 02:54:26.794763 (+ 56us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:26.794855 (+ 92us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:26.794954 (+ 99us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:26.795140 (+ 186us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:26.795259 (+ 119us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:26.795304 (+ 45us) write_op.cc:312] APPLY: finished
0114 02:54:26.795487 (+ 183us) log.cc:844] Serialized 23 byte log entry
0114 02:54:26.795624 (+ 137us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:26.795679 (+ 55us) write_op.cc:454] Released schema lock
0114 02:54:26.795712 (+ 33us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":71422,"reported_tablets":26,"spinlock_wait_cycles":14922112,"child_traces":[["op",{"apply.queue_time_us":5548,"apply.run_cpu_time_us":1224,"apply.run_wall_time_us":1304,"num_ops":1,"prepare.queue_time_us":123,"prepare.run_cpu_time_us":1720,"prepare.run_wall_time_us":1720,"raft.queue_time_us":442,"raft.run_cpu_time_us":708,"raft.run_wall_time_us":705,"replication_time_us":7009}],["op",{"apply.queue_time_us":80,"apply.run_cpu_time_us":1168,"apply.run_wall_time_us":1168,"num_ops":1,"prepare.queue_time_us":52775,"prepare.run_cpu_time_us":2852,"prepare.run_wall_time_us":24368,"raft.queue_time_us":623,"raft.run_cpu_time_us":643,"raft.run_wall_time_us":8286,"replication_time_us":21233}]]}
I20250114 02:54:29.420387 1990 log.cc:1198] T 46360f726fad4534a2daf67581346fc1 P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/46360f726fad4534a2daf67581346fc1
I20250114 02:54:29.421648 1990 ts_tablet_manager.cc:1939] T 46360f726fad4534a2daf67581346fc1 P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
I20250114 02:54:29.416757 1992 ts_tablet_manager.cc:1939] T 1381e26725174b9c877075cf5444f566 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:29.428052 784 tablet_service.cc:1514] Processing DeleteTablet for tablet e15e6e52d7294aeeb4463b3a76022ec3 with delete_type TABLET_DATA_DELETED (replaced by tablet 6d741bfd00784d2d830be3ebe50dc7c4) from {username='slave'} at 127.0.0.1:53078
W20250114 02:54:29.429641 2140 tablet_replica.cc:1411] Time spent flushing the Log queue took a long time: real 0.204s user 0.000s sys 0.021s
I20250114 02:54:29.430158 1992 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 231) took 3646 ms. Trace:
W20250114 02:54:29.446708 898 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.65:39397, user_credentials={real_user=slave}} blocked reactor thread for 57672.5us
W20250114 02:54:29.424903 462 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.66:50663 (request call id 49) took 2957 ms (client timeout 1000 ms). Trace:
W20250114 02:54:29.454473 470 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.65:49805 (request call id 53) took 1624 ms (client timeout 1000 ms). Trace:
I20250114 02:54:29.448727 898 leader_election.cc:304] T 38fe8829d98547fb907ef78099df3d7f P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
I20250114 02:54:29.458218 2143 raft_consensus.cc:2798] T 38fe8829d98547fb907ef78099df3d7f P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:29.458688 2143 raft_consensus.cc:695] T 38fe8829d98547fb907ef78099df3d7f P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Becoming Leader. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Running, Role: LEADER
I20250114 02:54:29.459385 2143 consensus_queue.cc:237] T 38fe8829d98547fb907ef78099df3d7f P 525198593c8e47afaee54e74267b87e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:29.431293 785 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e858a8449444ef999b9bbef612249d3 with delete_type TABLET_DATA_DELETED (replaced by tablet 797d6cdf36704c98aa106aac87a945ca) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:29.464386 1992 rpcz_store.cc:276] 0114 02:54:25.783316 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:25.783426 (+ 110us) service_pool.cc:225] Handling call
0114 02:54:28.321745 (+2538319us) ts_tablet_manager.cc:1145] Deleting tablet 1381e26725174b9c877075cf5444f566
0114 02:54:28.321789 (+ 44us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:29.243927 (+922138us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:29.414694 (+170767us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:29.430087 (+ 15393us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2537824}
I20250114 02:54:29.466246 935 tablet_service.cc:1514] Processing DeleteTablet for tablet e15e6e52d7294aeeb4463b3a76022ec3 with delete_type TABLET_DATA_DELETED (replaced by tablet 6d741bfd00784d2d830be3ebe50dc7c4) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:29.477367 2169 tablet.cc:1721] T 044832851f804208b0a371eb0857dbf3 P 90d6c2c58bb54e0eafb365625ac33871: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250114 02:54:29.478967 471 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:33540 (request call id 280) took 1817 ms. Trace:
I20250114 02:54:29.481381 1990 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 249) took 3919 ms. Trace:
I20250114 02:54:29.479423 471 rpcz_store.cc:276] 0114 02:54:27.661826 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:27.732708 (+ 70882us) service_pool.cc:225] Handling call
0114 02:54:27.742125 (+ 9417us) spinlock_profiling.cc:243] Waited 8.65 ms on lock 0x7b54000316d4. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616cad50 00007f646174df2a 00007f64633f2dec 00007f64634ec986 00007f6461259644 00007f64612595d2 00007f6461259501 00007f646125947c 00007f6461258732 00007f645f512dfc 00007f645f512236
0114 02:54:27.744848 (+ 2723us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0114 02:54:27.745638 (+ 790us) catalog_manager.cc:3627] Apply alter schema
0114 02:54:27.746333 (+ 695us) catalog_manager.cc:3659] Acquired catalog manager lock
0114 02:54:27.746585 (+ 252us) catalog_manager.cc:3817] Updating metadata on disk
0114 02:54:27.892102 (+145517us) catalog_manager.cc:3848] Committing alterations to in-memory state
0114 02:54:29.478904 (+1586802us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:27.795460 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:27.795791 (+ 331us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:27.795812 (+ 21us) write_op.cc:435] Acquired schema lock
0114 02:54:27.795826 (+ 14us) tablet.cc:659] Decoding operations
0114 02:54:27.796049 (+ 223us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:27.796079 (+ 30us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:27.796102 (+ 23us) tablet.cc:682] Acquiring locks for 1 operations
0114 02:54:27.796277 (+ 175us) tablet.cc:698] Row locks acquired
0114 02:54:27.796288 (+ 11us) write_op.cc:260] PREPARE: finished
0114 02:54:27.796371 (+ 83us) write_op.cc:270] Start()
0114 02:54:27.815775 (+ 19404us) write_op.cc:276] Timestamp: P: 1736823267796301 usec, L: 0
0114 02:54:27.815803 (+ 28us) op_driver.cc:348] REPLICATION: starting
0114 02:54:27.823570 (+ 7767us) log.cc:844] Serialized 405 byte log entry
0114 02:54:27.886164 (+ 62594us) op_driver.cc:464] REPLICATION: finished
0114 02:54:27.889655 (+ 3491us) write_op.cc:301] APPLY: starting
0114 02:54:27.889719 (+ 64us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:27.889825 (+ 106us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:27.889942 (+ 117us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:27.890096 (+ 154us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:27.890267 (+ 171us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:27.890326 (+ 59us) write_op.cc:312] APPLY: finished
0114 02:54:27.890502 (+ 176us) log.cc:844] Serialized 23 byte log entry
0114 02:54:27.890629 (+ 127us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:27.890700 (+ 71us) write_op.cc:454] Released schema lock
0114 02:54:27.890736 (+ 36us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:27.890977 (+ 241us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:27.891015 (+ 38us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":19891968,"child_traces":[["op",{"apply.queue_time_us":3345,"apply.run_cpu_time_us":1280,"apply.run_wall_time_us":1280,"num_ops":1,"prepare.queue_time_us":30910,"prepare.run_cpu_time_us":2106,"prepare.run_wall_time_us":28312,"replication_time_us":70276}]]}
I20250114 02:54:29.481803 1990 rpcz_store.cc:276] 0114 02:54:25.561738 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:25.562358 (+ 620us) service_pool.cc:225] Handling call
0114 02:54:29.227597 (+3665239us) ts_tablet_manager.cc:1145] Deleting tablet 46360f726fad4534a2daf67581346fc1
0114 02:54:29.227652 (+ 55us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:29.371173 (+143521us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:29.414633 (+ 43460us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:29.481306 (+ 66673us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3580840}
I20250114 02:54:29.482975 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet 46360f726fad4534a2daf67581346fc1 (table default.Table_3e7cad29b0c0447ab9539b45ee482232 [id=f65443bbd53c47e184a6d2e9e99dd1b0]) successfully deleted
I20250114 02:54:29.483582 1990 tablet_replica.cc:331] T 1381e26725174b9c877075cf5444f566 P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:29.484687 1990 raft_consensus.cc:2238] T 1381e26725174b9c877075cf5444f566 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:29.485222 1990 raft_consensus.cc:2267] T 1381e26725174b9c877075cf5444f566 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:29.487607 1990 ts_tablet_manager.cc:1905] T 1381e26725174b9c877075cf5444f566 P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250114 02:54:29.449620 462 rpcz_store.cc:269] 0114 02:54:26.467394 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:26.467525 (+ 131us) service_pool.cc:225] Handling call
0114 02:54:29.424829 (+2957304us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:27.780377 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:27.781076 (+ 699us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:27.781103 (+ 27us) write_op.cc:435] Acquired schema lock
0114 02:54:27.781119 (+ 16us) tablet.cc:659] Decoding operations
0114 02:54:27.781385 (+ 266us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:27.781422 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:27.781446 (+ 24us) tablet.cc:682] Acquiring locks for 1 operations
0114 02:54:27.781590 (+ 144us) tablet.cc:698] Row locks acquired
0114 02:54:27.781605 (+ 15us) write_op.cc:260] PREPARE: finished
0114 02:54:27.781715 (+ 110us) write_op.cc:270] Start()
0114 02:54:27.781799 (+ 84us) write_op.cc:276] Timestamp: P: 1736823267781640 usec, L: 0
0114 02:54:27.781818 (+ 19us) op_driver.cc:348] REPLICATION: starting
0114 02:54:27.782587 (+ 769us) log.cc:844] Serialized 530 byte log entry
0114 02:54:27.792896 (+ 10309us) spinlock_profiling.cc:243] Waited 8.47 ms on lock 0x7b58000d01b0. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616cad50 00007f64616bfa94 00007f64616c49d1 00007f64616c4959 00007f64616c48e9 00007f64616c48b1 00007f64616c3a7d 000000000059ea04 000000000059e839 00007f645a7fc866 00007f645a7ffcc1
0114 02:54:27.885733 (+ 92837us) op_driver.cc:464] REPLICATION: finished
0114 02:54:27.886452 (+ 719us) write_op.cc:301] APPLY: starting
0114 02:54:27.886511 (+ 59us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:27.886610 (+ 99us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:27.886629 (+ 19us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:27.886809 (+ 180us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:27.886946 (+ 137us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:27.886996 (+ 50us) write_op.cc:312] APPLY: finished
0114 02:54:27.887184 (+ 188us) log.cc:844] Serialized 23 byte log entry
0114 02:54:27.887343 (+ 159us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:27.887407 (+ 64us) write_op.cc:454] Released schema lock
0114 02:54:27.887439 (+ 32us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:27.887661 (+ 222us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:27.887688 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":152212,"reported_tablets":26,"child_traces":[["op",{"apply.queue_time_us":575,"apply.run_cpu_time_us":1165,"apply.run_wall_time_us":1164,"num_ops":1,"prepare.queue_time_us":41527,"prepare.run_cpu_time_us":1953,"prepare.run_wall_time_us":2488,"raft.queue_time_us":244,"raft.run_cpu_time_us":784,"raft.run_wall_time_us":9919,"replication_time_us":103815,"spinlock_wait_cycles":19491712}]]}
I20250114 02:54:29.488432 1104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "719188f639724851a5fc2b37cb60fdbb" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:29.493268 1104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c86562688d5b4fffb71cbdf7be0afc37" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
W20250114 02:54:29.496888 465 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.66:50663 (request call id 50) took 1379 ms (client timeout 1000 ms). Trace:
I20250114 02:54:29.511442 1076 tablet_service.cc:1514] Processing DeleteTablet for tablet 46360f726fad4534a2daf67581346fc1 with delete_type TABLET_DATA_DELETED (replaced by tablet be7aad41ccdd4ef9bcfe8a985155708f) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:29.513293 1965 ts_tablet_manager.cc:1905] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250114 02:54:29.514581 898 leader_election.cc:343] T c86562688d5b4fffb71cbdf7be0afc37 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:29.516477 1086 tablet_service.cc:1514] Processing DeleteTablet for tablet e15e6e52d7294aeeb4463b3a76022ec3 with delete_type TABLET_DATA_DELETED (replaced by tablet 6d741bfd00784d2d830be3ebe50dc7c4) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:29.517702 468 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_bcf55148b6bc413e8d4e8e31996ff32b" } modify_external_catalogs: true
I20250114 02:54:29.518265 468 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_bcf55148b6bc413e8d4e8e31996ff32b" } modify_external_catalogs: true
I20250114 02:54:29.531782 1992 tablet_replica.cc:331] T 5ac7708550954a209e2a8d217c3b39bd P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:29.532841 1992 raft_consensus.cc:2238] T 5ac7708550954a209e2a8d217c3b39bd P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 LEADER]: Raft consensus shutting down.
I20250114 02:54:29.533898 1992 raft_consensus.cc:2267] T 5ac7708550954a209e2a8d217c3b39bd P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:29.533986 2162 raft_consensus.cc:491] T df045d80d8ce4ff194a6d2039a44bce6 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:29.534204 1423 webserver.cc:458] Webserver started at http://127.0.107.124:36399/ using document root <none> and password file <none>
I20250114 02:54:29.534425 2162 raft_consensus.cc:513] T df045d80d8ce4ff194a6d2039a44bce6 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:29.536056 1423 fs_manager.cc:362] Metadata directory not provided
I20250114 02:54:29.536340 1423 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 02:54:29.536445 2162 leader_election.cc:290] T df045d80d8ce4ff194a6d2039a44bce6 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:29.458148 1206 ts_tablet_manager.cc:1428] T ed7f575653c848d7b076c17c38c4a244 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.193s user 0.005s sys 0.002s
I20250114 02:54:29.458849 784 tablet_service.cc:1514] Processing DeleteTablet for tablet 46360f726fad4534a2daf67581346fc1 with delete_type TABLET_DATA_DELETED (replaced by tablet be7aad41ccdd4ef9bcfe8a985155708f) from {username='slave'} at 127.0.0.1:53078
W20250114 02:54:29.440331 751 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.66:44037, user_credentials={real_user=slave}} blocked reactor thread for 164752us
W20250114 02:54:29.537631 898 leader_election.cc:343] T 38fe8829d98547fb907ef78099df3d7f P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:29.539836 817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d05089eae2b74f9fb23a226d358a59da" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:29.540335 468 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet 25e85b0c5c9a4a3ca35a001359f21167
I20250114 02:54:29.540400 817 raft_consensus.cc:2463] T d05089eae2b74f9fb23a226d358a59da P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 0.
W20250114 02:54:29.541730 751 leader_election.cc:343] T 59a46bc865cc48df8e98598764fd1b54 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:29.542223 751 leader_election.cc:304] T 59a46bc865cc48df8e98598764fd1b54 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
W20250114 02:54:29.542418 464 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.67:34825 (request call id 49) took 2496 ms (client timeout 1000 ms). Trace:
I20250114 02:54:29.542752 2143 consensus_queue.cc:1035] T e15e6e52d7294aeeb4463b3a76022ec3 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:29.544463 1086 tablet_service.cc:1514] Processing DeleteTablet for tablet f0bd9bd5a17d418d8082ed0ebd1bf4cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:19 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:29.545063 898 leader_election.cc:304] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 525198593c8e47afaee54e74267b87e3; no voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:29.545092 1042 leader_election.cc:304] T d05089eae2b74f9fb23a226d358a59da P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters: 525198593c8e47afaee54e74267b87e3
I20250114 02:54:29.545783 469 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_17190cd416734d35abcc8c6b354e5bb1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:29.546131 1820 raft_consensus.cc:2798] T d05089eae2b74f9fb23a226d358a59da P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:29.546461 1820 raft_consensus.cc:491] T d05089eae2b74f9fb23a226d358a59da P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:29.546618 2016 raft_consensus.cc:2743] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:29.546775 1820 raft_consensus.cc:3054] T d05089eae2b74f9fb23a226d358a59da P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:29.549422 1990 ts_tablet_manager.cc:1918] T 1381e26725174b9c877075cf5444f566 P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250114 02:54:29.549777 1990 log.cc:1198] T 1381e26725174b9c877075cf5444f566 P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/1381e26725174b9c877075cf5444f566
I20250114 02:54:29.550522 1990 ts_tablet_manager.cc:1939] T 1381e26725174b9c877075cf5444f566 P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
W20250114 02:54:29.455153 470 rpcz_store.cc:269] 0114 02:54:27.829758 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:27.865497 (+ 35739us) service_pool.cc:225] Handling call
0114 02:54:28.968298 (+1102801us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:28.968324 (+ 26us) write_op.cc:454] Released schema lock
0114 02:54:29.454384 (+486060us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:28.953503 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:28.954029 (+ 526us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:28.954064 (+ 35us) write_op.cc:435] Acquired schema lock
0114 02:54:28.954087 (+ 23us) tablet.cc:659] Decoding operations
0114 02:54:28.954429 (+ 342us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:28.954495 (+ 66us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:28.954531 (+ 36us) tablet.cc:682] Acquiring locks for 1 operations
0114 02:54:28.954730 (+ 199us) tablet.cc:698] Row locks acquired
0114 02:54:28.954752 (+ 22us) write_op.cc:260] PREPARE: finished
0114 02:54:28.954889 (+ 137us) write_op.cc:270] Start()
0114 02:54:28.955001 (+ 112us) write_op.cc:276] Timestamp: P: 1736823268954827 usec, L: 0
0114 02:54:28.955027 (+ 26us) op_driver.cc:348] REPLICATION: starting
0114 02:54:28.955696 (+ 669us) log.cc:844] Serialized 530 byte log entry
0114 02:54:28.963659 (+ 7963us) op_driver.cc:464] REPLICATION: finished
0114 02:54:28.964060 (+ 401us) write_op.cc:301] APPLY: starting
0114 02:54:28.964112 (+ 52us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:28.964199 (+ 87us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:28.964313 (+ 114us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:28.964535 (+ 222us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:28.964679 (+ 144us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:28.964719 (+ 40us) write_op.cc:312] APPLY: finished
0114 02:54:28.964890 (+ 171us) log.cc:844] Serialized 23 byte log entry
0114 02:54:28.965025 (+ 135us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:28.965086 (+ 61us) write_op.cc:454] Released schema lock
0114 02:54:28.965114 (+ 28us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":744,"reported_tablets":32,"child_traces":[["op",{"apply.queue_time_us":269,"apply.run_cpu_time_us":1238,"apply.run_wall_time_us":1238,"mutex_wait_us":129,"num_ops":1,"prepare.queue_time_us":113,"prepare.run_cpu_time_us":2613,"prepare.run_wall_time_us":2734,"raft.queue_time_us":385,"raft.run_cpu_time_us":642,"raft.run_wall_time_us":836,"replication_time_us":8539}]]}
I20250114 02:54:29.552587 1990 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 255) took 3693 ms. Trace:
I20250114 02:54:29.552886 1990 rpcz_store.cc:276] 0114 02:54:25.859501 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:25.864876 (+ 5375us) service_pool.cc:225] Handling call
0114 02:54:29.483395 (+3618519us) ts_tablet_manager.cc:1145] Deleting tablet 1381e26725174b9c877075cf5444f566
0114 02:54:29.483459 (+ 64us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:29.508694 (+ 25235us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:29.549220 (+ 40526us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:29.552535 (+ 3315us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3617984}
I20250114 02:54:29.554953 2093 raft_consensus.cc:2743] T 59a46bc865cc48df8e98598764fd1b54 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:29.554931 462 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_dda4f31205ab46c7a3af70bbb8037ec8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:29.555552 1990 tablet_replica.cc:331] T 5ac7708550954a209e2a8d217c3b39bd P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:29.557742 1990 raft_consensus.cc:2238] T 5ac7708550954a209e2a8d217c3b39bd P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:29.558229 1990 raft_consensus.cc:2267] T 5ac7708550954a209e2a8d217c3b39bd P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:29.559034 1820 raft_consensus.cc:513] T d05089eae2b74f9fb23a226d358a59da P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:29.561316 935 tablet_service.cc:1514] Processing DeleteTablet for tablet 25e85b0c5c9a4a3ca35a001359f21167 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:29 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:29.562184 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d05089eae2b74f9fb23a226d358a59da" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
W20250114 02:54:29.563349 1040 leader_election.cc:343] T d05089eae2b74f9fb23a226d358a59da P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:29.565304 815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d05089eae2b74f9fb23a226d358a59da" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:29.565884 815 raft_consensus.cc:3054] T d05089eae2b74f9fb23a226d358a59da P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
W20250114 02:54:29.569862 898 leader_election.cc:343] T 719188f639724851a5fc2b37cb60fdbb P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:29.572801 815 raft_consensus.cc:2463] T d05089eae2b74f9fb23a226d358a59da P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1.
I20250114 02:54:29.486656 785 tablet_service.cc:1514] Processing DeleteTablet for tablet f0bd9bd5a17d418d8082ed0ebd1bf4cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:19 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:29.487483 786 tablet_service.cc:1514] Processing DeleteTablet for tablet 321b9ad6d1ac4d1c80c7a01faf2015af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:23 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:29.488312 787 tablet_service.cc:1514] Processing DeleteTablet for tablet ca986f81ed6848a89588366f3f400eec with delete_type TABLET_DATA_DELETED (replaced by tablet a2d68eccaaa64511a9ae36f2b07a7b59) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:29.489053 794 tablet_service.cc:1514] Processing DeleteTablet for tablet c86562688d5b4fffb71cbdf7be0afc37 with delete_type TABLET_DATA_DELETED (replaced by tablet e7ca0bd1dbd542de9ee11e6c72522ae0) from {username='slave'} at 127.0.0.1:53078
W20250114 02:54:29.485064 753 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.67:45749, user_credentials={real_user=slave}} blocked reactor thread for 87585.5us
I20250114 02:54:29.575140 1102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "719188f639724851a5fc2b37cb60fdbb" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
W20250114 02:54:29.575336 898 leader_election.cc:343] T c86562688d5b4fffb71cbdf7be0afc37 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:29.577128 1102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c86562688d5b4fffb71cbdf7be0afc37" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:29.578949 1990 ts_tablet_manager.cc:1905] T 5ac7708550954a209e2a8d217c3b39bd P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:29.585199 1102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df045d80d8ce4ff194a6d2039a44bce6" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:29.556782 471 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_ceb42359708442c294b3f50aa6eadcc6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:29.561691 469 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_17190cd416734d35abcc8c6b354e5bb1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:29.588451 1206 tablet_bootstrap.cc:492] T fc406eab968745d893f1a8538ff61f37 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:29.596552 1423 fs_manager.cc:714] Time spent opening directory manager: real 0.052s user 0.005s sys 0.004s
I20250114 02:54:29.598803 1205 tablet_bootstrap.cc:700] T a52d6749ff834b62a8e20a2f45935240 P 525198593c8e47afaee54e74267b87e3: Time spent opening tablet: real 0.131s user 0.003s sys 0.000s
I20250114 02:54:29.602576 1205 tablet_bootstrap.cc:654] T a52d6749ff834b62a8e20a2f45935240 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:29.605937 1820 leader_election.cc:290] T d05089eae2b74f9fb23a226d358a59da P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Requested vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:29.608745 2109 tablet.cc:1721] T 044832851f804208b0a371eb0857dbf3 P 819f9e3412e54d44abb11d8b3ca877c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250114 02:54:29.610941 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "719188f639724851a5fc2b37cb60fdbb" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
I20250114 02:54:29.611191 958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c86562688d5b4fffb71cbdf7be0afc37" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
I20250114 02:54:29.611678 962 raft_consensus.cc:2388] T 719188f639724851a5fc2b37cb60fdbb P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 819f9e3412e54d44abb11d8b3ca877c2 in current term 1: Already voted for candidate 525198593c8e47afaee54e74267b87e3 in this term.
I20250114 02:54:29.611809 958 raft_consensus.cc:2388] T c86562688d5b4fffb71cbdf7be0afc37 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 819f9e3412e54d44abb11d8b3ca877c2 in current term 1: Already voted for candidate 525198593c8e47afaee54e74267b87e3 in this term.
I20250114 02:54:29.611887 2169 tablet.cc:1721] T 9bfd8fd6d2104fb4a6484cb8c98981bd P 90d6c2c58bb54e0eafb365625ac33871: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250114 02:54:29.616909 468 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:33540 (request call id 287) took 1544 ms. Trace:
I20250114 02:54:29.617601 1086 tablet_service.cc:1514] Processing DeleteTablet for tablet 25e85b0c5c9a4a3ca35a001359f21167 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:29 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:29.617257 468 rpcz_store.cc:276] 0114 02:54:28.072934 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:29.409942 (+1337008us) service_pool.cc:225] Handling call
0114 02:54:29.517615 (+107673us) spinlock_profiling.cc:243] Waited 107 ms on lock 0x7b54000316d4. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616cad50 00007f646174df2a 00007f64633f2dec 00007f64634ec616 00007f6461254b44 00007f6461254ad2 00007f6461254a01 00007f646125497c 00007f6461253c32 00007f645f512dfc 00007f645f512236
0114 02:54:29.518731 (+ 1116us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0114 02:54:29.519187 (+ 456us) catalog_manager.cc:2753] Modifying in-memory table state
0114 02:54:29.519336 (+ 149us) catalog_manager.cc:2762] Locking tablets
0114 02:54:29.519840 (+ 504us) catalog_manager.cc:2776] Removing table and tablets from system table
0114 02:54:29.539697 (+ 19857us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:29.539722 (+ 25us) write_op.cc:454] Released schema lock
0114 02:54:29.539941 (+ 219us) catalog_manager.cc:2794] Removing table from by-name map
0114 02:54:29.540128 (+ 187us) catalog_manager.cc:2810] Committing in-memory state
0114 02:54:29.540255 (+ 127us) catalog_manager.cc:2814] Aborting table tasks
0114 02:54:29.616852 (+ 76597us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:29.521725 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:29.522152 (+ 427us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:29.522179 (+ 27us) write_op.cc:435] Acquired schema lock
0114 02:54:29.522195 (+ 16us) tablet.cc:659] Decoding operations
0114 02:54:29.522550 (+ 355us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:29.522594 (+ 44us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:29.522626 (+ 32us) tablet.cc:682] Acquiring locks for 2 operations
0114 02:54:29.522919 (+ 293us) tablet.cc:698] Row locks acquired
0114 02:54:29.522937 (+ 18us) write_op.cc:260] PREPARE: finished
0114 02:54:29.523067 (+ 130us) write_op.cc:270] Start()
0114 02:54:29.523157 (+ 90us) write_op.cc:276] Timestamp: P: 1736823269523027 usec, L: 0
0114 02:54:29.523175 (+ 18us) op_driver.cc:348] REPLICATION: starting
0114 02:54:29.523682 (+ 507us) log.cc:844] Serialized 762 byte log entry
0114 02:54:29.532194 (+ 8512us) op_driver.cc:464] REPLICATION: finished
0114 02:54:29.532513 (+ 319us) write_op.cc:301] APPLY: starting
0114 02:54:29.532584 (+ 71us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:29.532707 (+ 123us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:29.532795 (+ 88us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:29.533044 (+ 249us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:29.533175 (+ 131us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0114 02:54:29.533216 (+ 41us) write_op.cc:312] APPLY: finished
0114 02:54:29.533436 (+ 220us) log.cc:844] Serialized 29 byte log entry
0114 02:54:29.533571 (+ 135us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:29.533633 (+ 62us) write_op.cc:454] Released schema lock
0114 02:54:29.533662 (+ 29us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":245994240,"child_traces":[["op",{"apply.queue_time_us":157,"apply.run_cpu_time_us":1297,"apply.run_wall_time_us":1297,"num_ops":2,"prepare.queue_time_us":105,"prepare.run_cpu_time_us":2234,"prepare.run_wall_time_us":2235,"raft.queue_time_us":161,"raft.run_cpu_time_us":665,"raft.run_wall_time_us":15548,"replication_time_us":8923}]]}
I20250114 02:54:29.621906 468 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_2c307103a2e8474b9c7e18b5691f61d4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:29.623642 468 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2c307103a2e8474b9c7e18b5691f61d4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:29.625162 2174 tablet.cc:1721] T 9bfd8fd6d2104fb4a6484cb8c98981bd P 525198593c8e47afaee54e74267b87e3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250114 02:54:29.628571 811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "719188f639724851a5fc2b37cb60fdbb" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:29.629004 1992 ts_tablet_manager.cc:1905] T 5ac7708550954a209e2a8d217c3b39bd P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:29.629317 811 raft_consensus.cc:2388] T 719188f639724851a5fc2b37cb60fdbb P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 525198593c8e47afaee54e74267b87e3 in current term 1: Already voted for candidate 819f9e3412e54d44abb11d8b3ca877c2 in this term.
I20250114 02:54:29.648840 2179 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
W20250114 02:54:29.650282 898 leader_election.cc:343] T df045d80d8ce4ff194a6d2039a44bce6 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:29.651091 2140 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:55068 (request call id 290) took 1564 ms. Trace:
I20250114 02:54:29.653224 1423 fs_manager.cc:730] Time spent opening block manager: real 0.051s user 0.004s sys 0.003s
W20250114 02:54:29.497309 465 rpcz_store.cc:269] 0114 02:54:28.117840 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:28.733873 (+616033us) service_pool.cc:225] Handling call
0114 02:54:29.032083 (+298210us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:29.032107 (+ 24us) write_op.cc:454] Released schema lock
0114 02:54:29.238798 (+206691us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:29.238823 (+ 25us) write_op.cc:454] Released schema lock
0114 02:54:29.496825 (+258002us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:29.013719 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:29.014108 (+ 389us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:29.014132 (+ 24us) write_op.cc:435] Acquired schema lock
0114 02:54:29.014147 (+ 15us) tablet.cc:659] Decoding operations
0114 02:54:29.014413 (+ 266us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:29.014452 (+ 39us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:29.014476 (+ 24us) tablet.cc:682] Acquiring locks for 1 operations
0114 02:54:29.014618 (+ 142us) tablet.cc:698] Row locks acquired
0114 02:54:29.014632 (+ 14us) write_op.cc:260] PREPARE: finished
0114 02:54:29.014736 (+ 104us) write_op.cc:270] Start()
0114 02:54:29.014818 (+ 82us) write_op.cc:276] Timestamp: P: 1736823269014678 usec, L: 0
0114 02:54:29.014835 (+ 17us) op_driver.cc:348] REPLICATION: starting
0114 02:54:29.015385 (+ 550us) log.cc:844] Serialized 530 byte log entry
0114 02:54:29.024109 (+ 8724us) op_driver.cc:464] REPLICATION: finished
0114 02:54:29.024478 (+ 369us) write_op.cc:301] APPLY: starting
0114 02:54:29.024530 (+ 52us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:29.024615 (+ 85us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:29.024707 (+ 92us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:29.024860 (+ 153us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:29.024975 (+ 115us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:29.025015 (+ 40us) write_op.cc:312] APPLY: finished
0114 02:54:29.025184 (+ 169us) log.cc:844] Serialized 23 byte log entry
0114 02:54:29.025371 (+ 187us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:29.025429 (+ 58us) write_op.cc:454] Released schema lock
0114 02:54:29.025457 (+ 28us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0114 02:54:29.217432 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:29.217839 (+ 407us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:29.217864 (+ 25us) write_op.cc:435] Acquired schema lock
0114 02:54:29.217886 (+ 22us) tablet.cc:659] Decoding operations
0114 02:54:29.218198 (+ 312us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:29.218272 (+ 74us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:29.218315 (+ 43us) tablet.cc:682] Acquiring locks for 1 operations
0114 02:54:29.218518 (+ 203us) tablet.cc:698] Row locks acquired
0114 02:54:29.218541 (+ 23us) write_op.cc:260] PREPARE: finished
0114 02:54:29.218702 (+ 161us) write_op.cc:270] Start()
0114 02:54:29.218818 (+ 116us) write_op.cc:276] Timestamp: P: 1736823269218608 usec, L: 0
0114 02:54:29.218845 (+ 27us) op_driver.cc:348] REPLICATION: starting
0114 02:54:29.219521 (+ 676us) log.cc:844] Serialized 356 byte log entry
0114 02:54:29.229614 (+ 10093us) op_driver.cc:464] REPLICATION: finished
0114 02:54:29.230789 (+ 1175us) write_op.cc:301] APPLY: starting
0114 02:54:29.230845 (+ 56us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:29.230933 (+ 88us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:29.231032 (+ 99us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:29.231184 (+ 152us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:29.231321 (+ 137us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:29.231373 (+ 52us) write_op.cc:312] APPLY: finished
0114 02:54:29.232107 (+ 734us) log.cc:844] Serialized 23 byte log entry
0114 02:54:29.232277 (+ 170us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:29.232342 (+ 65us) write_op.cc:454] Released schema lock
0114 02:54:29.232371 (+ 29us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":58075,"reported_tablets":29,"child_traces":[["op",{"apply.queue_time_us":242,"apply.run_cpu_time_us":1125,"apply.run_wall_time_us":1125,"num_ops":1,"prepare.queue_time_us":106,"prepare.run_cpu_time_us":1982,"prepare.run_wall_time_us":1982,"raft.queue_time_us":457,"raft.run_cpu_time_us":629,"raft.run_wall_time_us":629,"replication_time_us":9203}],["op",{"apply.queue_time_us":1021,"apply.run_cpu_time_us":1742,"apply.run_wall_time_us":1742,"num_ops":1,"prepare.queue_time_us":129,"prepare.run_cpu_time_us":2435,"prepare.run_wall_time_us":2434,"raft.queue_time_us":454,"raft.run_cpu_time_us":1102,"raft.run_wall_time_us":1101,"replication_time_us":10673}]]}
I20250114 02:54:29.667873 813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c86562688d5b4fffb71cbdf7be0afc37" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:29.653673 1423 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-2/data,/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-2/wal
uuid: "87b3db09ac87436e9a60d6de5fedaff5"
format_stamp: "Formatted at 2025-01-14 02:53:59 on dist-test-slave-36vg"
I20250114 02:54:29.668701 813 raft_consensus.cc:2388] T c86562688d5b4fffb71cbdf7be0afc37 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 525198593c8e47afaee54e74267b87e3 in current term 1: Already voted for candidate 819f9e3412e54d44abb11d8b3ca877c2 in this term.
I20250114 02:54:29.668767 1423 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 02:54:29.658023 465 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_4be1140a3b21462e9be7fe47af15dd09" } modify_external_catalogs: true
W20250114 02:54:29.664436 471 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ceb42359708442c294b3f50aa6eadcc6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250114 02:54:29.666452 462 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dda4f31205ab46c7a3af70bbb8037ec8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:29.679952 469 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33540 (request call id 289) took 1549 ms. Trace:
I20250114 02:54:29.681964 465 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_4be1140a3b21462e9be7fe47af15dd09" } modify_external_catalogs: true
I20250114 02:54:29.684222 786 tablet_service.cc:1514] Processing DeleteTablet for tablet 25e85b0c5c9a4a3ca35a001359f21167 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:29 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:29.684957 1990 ts_tablet_manager.cc:1918] T 5ac7708550954a209e2a8d217c3b39bd P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:29.685341 1990 log.cc:1198] T 5ac7708550954a209e2a8d217c3b39bd P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/5ac7708550954a209e2a8d217c3b39bd
I20250114 02:54:29.686024 1990 ts_tablet_manager.cc:1939] T 5ac7708550954a209e2a8d217c3b39bd P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
I20250114 02:54:29.687755 1990 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 259) took 3168 ms. Trace:
I20250114 02:54:29.691637 2188 tablet.cc:1721] T 92cd8eccacf34ab7a66b3bbc69df3697 P 90d6c2c58bb54e0eafb365625ac33871: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250114 02:54:29.710867 2184 tablet.cc:1721] T 92cd8eccacf34ab7a66b3bbc69df3697 P 525198593c8e47afaee54e74267b87e3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250114 02:54:29.712889 1207 tablet_bootstrap.cc:492] T 38fe8829d98547fb907ef78099df3d7f P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:29.713347 1207 ts_tablet_manager.cc:1397] T 38fe8829d98547fb907ef78099df3d7f P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.443s user 0.008s sys 0.002s
I20250114 02:54:29.715675 1207 raft_consensus.cc:357] T 38fe8829d98547fb907ef78099df3d7f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:29.716933 1042 leader_election.cc:304] T d05089eae2b74f9fb23a226d358a59da P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters: 525198593c8e47afaee54e74267b87e3
I20250114 02:54:29.718231 2133 raft_consensus.cc:2798] T d05089eae2b74f9fb23a226d358a59da P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:29.718639 2133 raft_consensus.cc:695] T d05089eae2b74f9fb23a226d358a59da P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Becoming Leader. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Running, Role: LEADER
I20250114 02:54:29.719463 2133 consensus_queue.cc:237] T d05089eae2b74f9fb23a226d358a59da P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:29.725332 2169 tablet.cc:1721] T 15704785f56e43a3af69f181c75877ce P 90d6c2c58bb54e0eafb365625ac33871: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250114 02:54:29.730320 2190 tablet.cc:1721] T 15704785f56e43a3af69f181c75877ce P 525198593c8e47afaee54e74267b87e3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250114 02:54:29.734642 753 leader_election.cc:343] T 719188f639724851a5fc2b37cb60fdbb P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:29.735342 813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df045d80d8ce4ff194a6d2039a44bce6" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:29.746105 753 leader_election.cc:304] T 719188f639724851a5fc2b37cb60fdbb P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
W20250114 02:54:29.753434 753 leader_election.cc:343] T c86562688d5b4fffb71cbdf7be0afc37 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:29.680397 469 rpcz_store.cc:276] 0114 02:54:28.130824 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:29.485141 (+1354317us) service_pool.cc:225] Handling call
0114 02:54:29.545596 (+ 60455us) spinlock_profiling.cc:243] Waited 31.6 ms on lock 0x7b54000316d4. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616cad50 00007f646174df2a 00007f64633f2dec 00007f64634ec306 00007f6461250044 00007f646124ffd2 00007f646124ff01 00007f646124fe7c 00007f646124f132 00007f645f512dfc 00007f645f512236
0114 02:54:29.587409 (+ 41813us) catalog_manager.cc:2076] Acquired catalog manager lock
0114 02:54:29.588912 (+ 1503us) catalog_manager.cc:2122] Created new table and tablet info
0114 02:54:29.672432 (+ 83520us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:29.672468 (+ 36us) write_op.cc:454] Released schema lock
0114 02:54:29.672720 (+ 252us) catalog_manager.cc:2182] Wrote table and tablets to system table
0114 02:54:29.679578 (+ 6858us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0114 02:54:29.679897 (+ 319us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:29.590454 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:29.590947 (+ 493us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:29.590982 (+ 35us) write_op.cc:435] Acquired schema lock
0114 02:54:29.591005 (+ 23us) tablet.cc:659] Decoding operations
0114 02:54:29.591444 (+ 439us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:29.591501 (+ 57us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:29.591536 (+ 35us) tablet.cc:682] Acquiring locks for 2 operations
0114 02:54:29.591812 (+ 276us) tablet.cc:698] Row locks acquired
0114 02:54:29.591835 (+ 23us) write_op.cc:260] PREPARE: finished
0114 02:54:29.591975 (+ 140us) write_op.cc:270] Start()
0114 02:54:29.592090 (+ 115us) write_op.cc:276] Timestamp: P: 1736823269591923 usec, L: 0
0114 02:54:29.592115 (+ 25us) op_driver.cc:348] REPLICATION: starting
0114 02:54:29.592775 (+ 660us) log.cc:844] Serialized 433 byte log entry
0114 02:54:29.602380 (+ 9605us) op_driver.cc:464] REPLICATION: finished
0114 02:54:29.602660 (+ 280us) write_op.cc:301] APPLY: starting
0114 02:54:29.602729 (+ 69us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:29.602861 (+ 132us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:29.602997 (+ 136us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:29.603501 (+ 504us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:29.603676 (+ 175us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 02:54:29.603754 (+ 78us) write_op.cc:312] APPLY: finished
0114 02:54:29.603973 (+ 219us) log.cc:844] Serialized 29 byte log entry
0114 02:54:29.604159 (+ 186us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:29.604262 (+ 103us) write_op.cc:454] Released schema lock
0114 02:54:29.604296 (+ 34us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":72743168,"child_traces":[["op",{"apply.queue_time_us":121,"apply.run_cpu_time_us":1828,"apply.run_wall_time_us":1829,"num_ops":2,"prepare.queue_time_us":156,"prepare.run_cpu_time_us":2636,"prepare.run_wall_time_us":2637,"raft.queue_time_us":408,"raft.run_cpu_time_us":628,"raft.run_wall_time_us":628,"replication_time_us":10141}]]}
I20250114 02:54:29.756489 469 master_service.cc:946] ReplaceTablet: received request to replace tablet 3161f75bc4e84000af022faa9804c810 from {username='slave'} at 127.0.0.1:33540
W20250114 02:54:29.542733 464 rpcz_store.cc:269] 0114 02:54:27.046022 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:27.183375 (+137353us) service_pool.cc:225] Handling call
0114 02:54:29.542354 (+2358979us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:28.824578 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:28.825008 (+ 430us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:28.825036 (+ 28us) write_op.cc:435] Acquired schema lock
0114 02:54:28.825053 (+ 17us) tablet.cc:659] Decoding operations
0114 02:54:28.825382 (+ 329us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:28.825427 (+ 45us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:28.825454 (+ 27us) tablet.cc:682] Acquiring locks for 1 operations
0114 02:54:28.825622 (+ 168us) tablet.cc:698] Row locks acquired
0114 02:54:28.825641 (+ 19us) write_op.cc:260] PREPARE: finished
0114 02:54:28.825765 (+ 124us) write_op.cc:270] Start()
0114 02:54:28.825859 (+ 94us) write_op.cc:276] Timestamp: P: 1736823268825686 usec, L: 0
0114 02:54:28.825878 (+ 19us) op_driver.cc:348] REPLICATION: starting
0114 02:54:28.832551 (+ 6673us) log.cc:844] Serialized 530 byte log entry
0114 02:54:28.841032 (+ 8481us) op_driver.cc:464] REPLICATION: finished
0114 02:54:28.841246 (+ 214us) write_op.cc:301] APPLY: starting
0114 02:54:28.841298 (+ 52us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:28.841380 (+ 82us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:28.841456 (+ 76us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:28.841639 (+ 183us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:28.841753 (+ 114us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:28.841795 (+ 42us) write_op.cc:312] APPLY: finished
0114 02:54:28.841949 (+ 154us) log.cc:844] Serialized 23 byte log entry
0114 02:54:28.842081 (+ 132us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:28.842133 (+ 52us) write_op.cc:454] Released schema lock
0114 02:54:28.842163 (+ 30us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0114 02:54:29.249450 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:29.249918 (+ 468us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:29.249950 (+ 32us) write_op.cc:435] Acquired schema lock
0114 02:54:29.249974 (+ 24us) tablet.cc:659] Decoding operations
0114 02:54:29.250285 (+ 311us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:29.250333 (+ 48us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:29.250367 (+ 34us) tablet.cc:682] Acquiring locks for 1 operations
0114 02:54:29.250539 (+ 172us) tablet.cc:698] Row locks acquired
0114 02:54:29.250559 (+ 20us) write_op.cc:260] PREPARE: finished
0114 02:54:29.250687 (+ 128us) write_op.cc:270] Start()
0114 02:54:29.250779 (+ 92us) write_op.cc:276] Timestamp: P: 1736823269250601 usec, L: 0
0114 02:54:29.250800 (+ 21us) op_driver.cc:348] REPLICATION: starting
0114 02:54:29.301619 (+ 50819us) spinlock_profiling.cc:243] Waited 49.7 ms on lock 0x7b5400032198. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64624790f4 00007f6461719389 00007f64616d51f1 00007f64616d4eca 00007f6461740cd5 00007f6461741179 00007f6461bad3b1 00007f6461bacb88 00007f6461bb1821 00007f6461bb17d9 00007f6461bb1769
0114 02:54:29.301955 (+ 336us) log.cc:844] Serialized 356 byte log entry
0114 02:54:29.311269 (+ 9314us) op_driver.cc:464] REPLICATION: finished
0114 02:54:29.311718 (+ 449us) write_op.cc:301] APPLY: starting
0114 02:54:29.311778 (+ 60us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:29.311889 (+ 111us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:29.311915 (+ 26us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:29.312127 (+ 212us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:29.313196 (+ 1069us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:29.313269 (+ 73us) write_op.cc:312] APPLY: finished
0114 02:54:29.313452 (+ 183us) log.cc:844] Serialized 23 byte log entry
0114 02:54:29.313593 (+ 141us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:29.313650 (+ 57us) write_op.cc:454] Released schema lock
0114 02:54:29.313678 (+ 28us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:29.324942 (+ 11264us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:29.324977 (+ 35us) write_op.cc:454] Released schema lock
0114 02:54:29.504652 (+179675us) spinlock_profiling.cc:243] Waited 158 ms on lock 0x7b58000d01b0. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616cad50 00007f64616bfa94 00007f64616c49d1 00007f64616c4959 00007f64616c48e9 00007f64616c48b1 00007f64616c3a7d 000000000059ea04 000000000059e839 00007f645a7fc866 00007f645a7ffcc1
Metrics: {"mutex_wait_us":96965,"reported_tablets":30,"child_traces":[["op",{"apply.queue_time_us":77,"apply.run_cpu_time_us":1086,"apply.run_wall_time_us":1083,"num_ops":1,"prepare.queue_time_us":6348,"prepare.run_cpu_time_us":2256,"prepare.run_wall_time_us":8327,"raft.queue_time_us":473,"raft.run_cpu_time_us":660,"raft.run_wall_time_us":13318,"replication_time_us":15067}],["op",{"apply.queue_time_us":279,"apply.run_cpu_time_us":1262,"apply.run_wall_time_us":2127,"num_ops":1,"prepare.queue_time_us":195,"prepare.run_cpu_time_us":4015,"prepare.run_wall_time_us":75474,"raft.queue_time_us":141,"raft.run_cpu_time_us":2679,"raft.run_wall_time_us":203092,"replication_time_us":60352,"spinlock_wait_cycles":477070592}]]}
I20250114 02:54:29.761363 1086 tablet_service.cc:1514] Processing DeleteTablet for tablet 84738a6c78b24c1fb64013048763b18f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:23 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:29.688105 1990 rpcz_store.cc:276] 0114 02:54:26.519159 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:26.519360 (+ 201us) service_pool.cc:225] Handling call
0114 02:54:29.555404 (+3036044us) ts_tablet_manager.cc:1145] Deleting tablet 5ac7708550954a209e2a8d217c3b39bd
0114 02:54:29.555441 (+ 37us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:29.606845 (+ 51404us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:29.684752 (+ 77907us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:29.687704 (+ 2952us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3025625}
I20250114 02:54:29.755658 2164 raft_consensus.cc:2743] T 719188f639724851a5fc2b37cb60fdbb P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250114 02:54:29.764453 1990 tablet_replica.cc:331] T de4ca83a69704a368a25954cdc1b9392 P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:29.764410 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet 5ac7708550954a209e2a8d217c3b39bd (table default.Table_ee814dd266e64286864db0eac9e11664 [id=13151866bf2b4f44b1dd5c92380a4d43]) successfully deleted
I20250114 02:54:29.765197 1990 raft_consensus.cc:2238] T de4ca83a69704a368a25954cdc1b9392 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:29.765108 753 leader_election.cc:304] T c86562688d5b4fffb71cbdf7be0afc37 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:29.765698 1990 raft_consensus.cc:2267] T de4ca83a69704a368a25954cdc1b9392 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:29.766281 1206 tablet_bootstrap.cc:654] T fc406eab968745d893f1a8538ff61f37 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:29.767537 1990 ts_tablet_manager.cc:1905] T de4ca83a69704a368a25954cdc1b9392 P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:29.768046 2164 raft_consensus.cc:2743] T c86562688d5b4fffb71cbdf7be0afc37 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250114 02:54:29.651548 2140 rpcz_store.cc:276] 0114 02:54:28.086482 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:28.087279 (+ 797us) service_pool.cc:225] Handling call
0114 02:54:29.650460 (+1563181us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:28.112684 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0114 02:54:28.117439 (+ 4755us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0114 02:54:28.117474 (+ 35us) alter_schema_op.cc:66] Acquired schema lock
0114 02:54:28.117520 (+ 46us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0114 02:54:28.120941 (+ 3421us) spinlock_profiling.cc:243] Waited 1.71 ms on lock 0x7b2c00005568. stack: 00007fb5ea74af63 00007fb5ea74ae60 00007fb5ea2c1584 00000000004fcd24 00000000004fccb9 00007fb5f09d2741 00007fb5f09e140a 00007fb5f09e1093 00007fb5f09e1031 00007fb5f1702042 00007fb5f1701e1d 00007fb5f1b2e073 00007fb5f1b2db88 00007fb5f1b32821 00007fb5f1b327d9 00007fb5f1b32769
0114 02:54:28.121021 (+ 80us) alter_schema_op.cc:128] START. Timestamp: 1736823268114253
0114 02:54:28.170608 (+ 49587us) op_driver.cc:348] REPLICATION: starting
0114 02:54:28.189007 (+ 18399us) log.cc:844] Serialized 173 byte log entry
0114 02:54:28.257823 (+ 68816us) spinlock_profiling.cc:243] Waited 3.94 ms on lock 0x7b58000f0bb0. stack: 00007fb5ea74af63 00007fb5ea74ae60 00007fb5ea2c1584 00000000004fcd24 00000000004fccb9 00007fb5f164bd50 00007fb5f1640a94 00007fb5f16459d1 00007fb5f1645959 00007fb5f16458e9 00007fb5f16458b1 00007fb5f1644a7d 000000000059ea04 000000000059e839 00007fb5ea77d866 00007fb5ea780cc1
0114 02:54:29.119464 (+861641us) op_driver.cc:464] REPLICATION: finished
0114 02:54:29.224889 (+105425us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0114 02:54:29.226089 (+ 1200us) log.cc:844] Serialized 5 byte log entry
0114 02:54:29.562299 (+336210us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:29.649352 (+ 87053us) log.cc:844] Serialized 14 byte log entry
0114 02:54:29.650326 (+ 974us) alter_schema_op.cc:73] Released schema lock
0114 02:54:29.650359 (+ 33us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":105211,"mutex_wait_us":16970,"prepare.queue_time_us":1744,"prepare.run_cpu_time_us":10111,"prepare.run_wall_time_us":163169,"raft.queue_time_us":18597,"raft.run_cpu_time_us":3716,"raft.run_wall_time_us":7819,"replication_time_us":931058,"spinlock_wait_cycles":13003520,"thread_start_us":347970,"threads_started":3,"wal-append.queue_time_us":229821,"wal-append.run_cpu_time_us":1767,"wal-append.run_wall_time_us":1001791}]]}
I20250114 02:54:29.778491 1086 tablet_service.cc:1514] Processing DeleteTablet for tablet 1381e26725174b9c877075cf5444f566 with delete_type TABLET_DATA_DELETED (replaced by tablet 9ffec1d1bb594fda83df8d94d4533d56) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:29.780172 1205 tablet_bootstrap.cc:492] T a52d6749ff834b62a8e20a2f45935240 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
W20250114 02:54:29.780236 466 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.67:34825 (request call id 50) took 1465 ms (client timeout 1000 ms). Trace:
I20250114 02:54:29.780814 1205 ts_tablet_manager.cc:1397] T a52d6749ff834b62a8e20a2f45935240 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.432s user 0.003s sys 0.009s
W20250114 02:54:29.780580 466 rpcz_store.cc:269] 0114 02:54:28.314963 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:28.945049 (+630086us) service_pool.cc:225] Handling call
0114 02:54:29.780135 (+835086us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9717,"reported_tablets":32}
I20250114 02:54:29.782613 466 master_service.cc:946] ReplaceTablet: received request to replace tablet d3fb28717c6f4844af84bb817fe7dfc4 from {username='slave'} at 127.0.0.1:33540
I20250114 02:54:29.783157 1207 raft_consensus.cc:383] T 38fe8829d98547fb907ef78099df3d7f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:29.789289 1207 raft_consensus.cc:738] T 38fe8829d98547fb907ef78099df3d7f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:29.770193 1965 ts_tablet_manager.cc:1918] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:29.805058 1965 log.cc:1198] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/f0bd9bd5a17d418d8082ed0ebd1bf4cc
I20250114 02:54:29.810446 464 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_80da487b7f04491888501948c55ed09f" } new_table_name: "default.Table_4364d1df16d149d2ba10469ce7288d4e" modify_external_catalogs: true
I20250114 02:54:29.814162 471 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33540 (request call id 288) took 1714 ms. Trace:
I20250114 02:54:29.814690 1207 consensus_queue.cc:260] T 38fe8829d98547fb907ef78099df3d7f P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:29.816764 1965 ts_tablet_manager.cc:1939] T f0bd9bd5a17d418d8082ed0ebd1bf4cc P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:29.814608 471 rpcz_store.cc:276] 0114 02:54:28.099801 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:29.484329 (+1384528us) service_pool.cc:225] Handling call
0114 02:54:29.548607 (+ 64278us) spinlock_profiling.cc:243] Waited 63.5 ms on lock 0x7b54000316d4. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616cad50 00007f646174df2a 00007f64633f2dec 00007f64634ec306 00007f6461250044 00007f646124ffd2 00007f646124ff01 00007f646124fe7c 00007f646124f132 00007f645f512dfc 00007f645f512236
0114 02:54:29.681630 (+133023us) catalog_manager.cc:2076] Acquired catalog manager lock
0114 02:54:29.693528 (+ 11898us) catalog_manager.cc:2122] Created new table and tablet info
0114 02:54:29.710973 (+ 17445us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:29.711002 (+ 29us) write_op.cc:454] Released schema lock
0114 02:54:29.711192 (+ 190us) catalog_manager.cc:2182] Wrote table and tablets to system table
0114 02:54:29.813830 (+102638us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0114 02:54:29.814087 (+ 257us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:29.694714 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:29.695103 (+ 389us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:29.695126 (+ 23us) write_op.cc:435] Acquired schema lock
0114 02:54:29.695139 (+ 13us) tablet.cc:659] Decoding operations
0114 02:54:29.695887 (+ 748us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:29.695925 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:29.695947 (+ 22us) tablet.cc:682] Acquiring locks for 2 operations
0114 02:54:29.696128 (+ 181us) tablet.cc:698] Row locks acquired
0114 02:54:29.696141 (+ 13us) write_op.cc:260] PREPARE: finished
0114 02:54:29.696306 (+ 165us) write_op.cc:270] Start()
0114 02:54:29.696392 (+ 86us) write_op.cc:276] Timestamp: P: 1736823269696239 usec, L: 0
0114 02:54:29.696409 (+ 17us) op_driver.cc:348] REPLICATION: starting
0114 02:54:29.696880 (+ 471us) log.cc:844] Serialized 433 byte log entry
0114 02:54:29.707310 (+ 10430us) op_driver.cc:464] REPLICATION: finished
0114 02:54:29.707824 (+ 514us) write_op.cc:301] APPLY: starting
0114 02:54:29.707885 (+ 61us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:29.708004 (+ 119us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:29.708148 (+ 144us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:29.708685 (+ 537us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:29.708860 (+ 175us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 02:54:29.708917 (+ 57us) write_op.cc:312] APPLY: finished
0114 02:54:29.709143 (+ 226us) log.cc:844] Serialized 29 byte log entry
0114 02:54:29.709312 (+ 169us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:29.709401 (+ 89us) write_op.cc:454] Released schema lock
0114 02:54:29.709441 (+ 40us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":145945344,"child_traces":[["op",{"apply.queue_time_us":336,"apply.run_cpu_time_us":1800,"apply.run_wall_time_us":1808,"num_ops":2,"prepare.queue_time_us":98,"prepare.run_cpu_time_us":2415,"prepare.run_wall_time_us":2416,"raft.queue_time_us":1652,"raft.run_cpu_time_us":763,"raft.run_wall_time_us":764,"replication_time_us":10781}]]}
I20250114 02:54:29.818492 462 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33540 (request call id 290) took 1611 ms. Trace:
I20250114 02:54:29.818912 462 rpcz_store.cc:276] 0114 02:54:28.207174 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:29.490289 (+1283115us) service_pool.cc:225] Handling call
0114 02:54:29.554755 (+ 64466us) spinlock_profiling.cc:243] Waited 58.2 ms on lock 0x7b54000316d4. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616cad50 00007f646174df2a 00007f64633f2dec 00007f64634ec306 00007f6461250044 00007f646124ffd2 00007f646124ff01 00007f646124fe7c 00007f646124f132 00007f645f512dfc 00007f645f512236
0114 02:54:29.681745 (+126990us) catalog_manager.cc:2076] Acquired catalog manager lock
0114 02:54:29.717864 (+ 36119us) catalog_manager.cc:2122] Created new table and tablet info
0114 02:54:29.750586 (+ 32722us) catalog_manager.cc:2182] Wrote table and tablets to system table
0114 02:54:29.818127 (+ 67541us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0114 02:54:29.818402 (+ 275us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:29.719062 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:29.719495 (+ 433us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:29.719524 (+ 29us) write_op.cc:435] Acquired schema lock
0114 02:54:29.719540 (+ 16us) tablet.cc:659] Decoding operations
0114 02:54:29.719833 (+ 293us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:29.719877 (+ 44us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:29.719901 (+ 24us) tablet.cc:682] Acquiring locks for 2 operations
0114 02:54:29.720103 (+ 202us) tablet.cc:698] Row locks acquired
0114 02:54:29.720123 (+ 20us) write_op.cc:260] PREPARE: finished
0114 02:54:29.720267 (+ 144us) write_op.cc:270] Start()
0114 02:54:29.720364 (+ 97us) write_op.cc:276] Timestamp: P: 1736823269720208 usec, L: 0
0114 02:54:29.720381 (+ 17us) op_driver.cc:348] REPLICATION: starting
0114 02:54:29.721648 (+ 1267us) log.cc:844] Serialized 433 byte log entry
0114 02:54:29.733970 (+ 12322us) op_driver.cc:464] REPLICATION: finished
0114 02:54:29.735788 (+ 1818us) write_op.cc:301] APPLY: starting
0114 02:54:29.735842 (+ 54us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:29.735940 (+ 98us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:29.736033 (+ 93us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:29.743159 (+ 7126us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:29.743292 (+ 133us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 02:54:29.743330 (+ 38us) write_op.cc:312] APPLY: finished
0114 02:54:29.743482 (+ 152us) log.cc:844] Serialized 29 byte log entry
0114 02:54:29.743607 (+ 125us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:29.743663 (+ 56us) write_op.cc:454] Released schema lock
0114 02:54:29.743693 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:29.743899 (+ 206us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:29.743921 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":133894528,"child_traces":[["op",{"apply.queue_time_us":1682,"apply.run_cpu_time_us":1404,"apply.run_wall_time_us":8082,"num_ops":2,"prepare.queue_time_us":92,"prepare.run_cpu_time_us":2134,"prepare.run_wall_time_us":7899,"raft.queue_time_us":3758,"raft.run_cpu_time_us":511,"raft.run_wall_time_us":512,"replication_time_us":13505}]]}
I20250114 02:54:29.824347 462 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_0943b1ca347042c9a6043006bc566701" } new_table_name: "default.Table_62251807af5845a89e89f8b2e003928f" modify_external_catalogs: true
I20250114 02:54:29.824909 2140 tablet.cc:1721] T 99b940d872ce4781907517c40fe08781 P 525198593c8e47afaee54e74267b87e3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250114 02:54:29.829406 2194 tablet.cc:1721] T 99b940d872ce4781907517c40fe08781 P 819f9e3412e54d44abb11d8b3ca877c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250114 02:54:29.832556 898 leader_election.cc:304] T 719188f639724851a5fc2b37cb60fdbb P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 525198593c8e47afaee54e74267b87e3; no voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:29.833503 2143 raft_consensus.cc:2743] T 719188f639724851a5fc2b37cb60fdbb P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250114 02:54:29.834371 468 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33540 (request call id 291) took 1594 ms. Trace:
I20250114 02:54:29.834582 898 leader_election.cc:304] T c86562688d5b4fffb71cbdf7be0afc37 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 525198593c8e47afaee54e74267b87e3; no voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:29.835424 2143 raft_consensus.cc:2743] T c86562688d5b4fffb71cbdf7be0afc37 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20250114 02:54:29.841193 898 leader_election.cc:343] T df045d80d8ce4ff194a6d2039a44bce6 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:29.841549 898 leader_election.cc:304] T df045d80d8ce4ff194a6d2039a44bce6 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 525198593c8e47afaee54e74267b87e3; no voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:29.842123 2016 raft_consensus.cc:2743] T df045d80d8ce4ff194a6d2039a44bce6 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:29.845710 471 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_e18d67a6f19d4259bee6f447deced18c" } modify_external_catalogs: true
I20250114 02:54:29.846314 471 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_e18d67a6f19d4259bee6f447deced18c" } modify_external_catalogs: true
I20250114 02:54:29.834738 468 rpcz_store.cc:276] 0114 02:54:28.240169 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:29.621365 (+1381196us) service_pool.cc:225] Handling call
0114 02:54:29.624060 (+ 2695us) catalog_manager.cc:2076] Acquired catalog manager lock
0114 02:54:29.766268 (+142208us) catalog_manager.cc:2122] Created new table and tablet info
0114 02:54:29.832945 (+ 66677us) catalog_manager.cc:2182] Wrote table and tablets to system table
0114 02:54:29.834072 (+ 1127us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0114 02:54:29.834316 (+ 244us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:29.767589 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:29.768028 (+ 439us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:29.768058 (+ 30us) write_op.cc:435] Acquired schema lock
0114 02:54:29.768099 (+ 41us) tablet.cc:659] Decoding operations
0114 02:54:29.768464 (+ 365us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:29.768511 (+ 47us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:29.768544 (+ 33us) tablet.cc:682] Acquiring locks for 2 operations
0114 02:54:29.768768 (+ 224us) tablet.cc:698] Row locks acquired
0114 02:54:29.768790 (+ 22us) write_op.cc:260] PREPARE: finished
0114 02:54:29.768920 (+ 130us) write_op.cc:270] Start()
0114 02:54:29.769021 (+ 101us) write_op.cc:276] Timestamp: P: 1736823269768850 usec, L: 0
0114 02:54:29.769042 (+ 21us) op_driver.cc:348] REPLICATION: starting
0114 02:54:29.790664 (+ 21622us) log.cc:844] Serialized 433 byte log entry
0114 02:54:29.792336 (+ 1672us) spinlock_profiling.cc:243] Waited 21 ms on lock 0x7b54000316d4. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616ca6d1 00007f6461741198 00007f6461bad3b1 00007f6461bacb88 00007f6461bb1821 00007f6461bb17d9 00007f6461bb1769 00007f6461bb1731 00007f6461bb095d 000000000059ea04 000000000059e839
0114 02:54:29.800689 (+ 8353us) op_driver.cc:464] REPLICATION: finished
0114 02:54:29.803608 (+ 2919us) write_op.cc:301] APPLY: starting
0114 02:54:29.803687 (+ 79us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:29.803801 (+ 114us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:29.803819 (+ 18us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:29.804226 (+ 407us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:29.804384 (+ 158us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 02:54:29.805062 (+ 678us) write_op.cc:312] APPLY: finished
0114 02:54:29.805255 (+ 193us) log.cc:844] Serialized 29 byte log entry
0114 02:54:29.805386 (+ 131us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:29.805452 (+ 66us) write_op.cc:454] Released schema lock
0114 02:54:29.805491 (+ 39us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:29.805712 (+ 221us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:29.805738 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2749,"apply.run_cpu_time_us":2039,"apply.run_wall_time_us":2094,"num_ops":2,"prepare.queue_time_us":168,"prepare.run_cpu_time_us":3290,"prepare.run_wall_time_us":26586,"raft.queue_time_us":94,"raft.run_cpu_time_us":651,"raft.run_wall_time_us":651,"replication_time_us":31537,"spinlock_wait_cycles":48355584}]]}
I20250114 02:54:29.868605 1205 raft_consensus.cc:357] T a52d6749ff834b62a8e20a2f45935240 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:29.869423 1205 raft_consensus.cc:383] T a52d6749ff834b62a8e20a2f45935240 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:29.869755 1205 raft_consensus.cc:738] T a52d6749ff834b62a8e20a2f45935240 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:29.870385 1205 consensus_queue.cc:260] T a52d6749ff834b62a8e20a2f45935240 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:29.873654 1207 ts_tablet_manager.cc:1428] T 38fe8829d98547fb907ef78099df3d7f P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.160s user 0.006s sys 0.001s
W20250114 02:54:29.876806 1042 consensus_peers.cc:487] T 84738a6c78b24c1fb64013048763b18f P 90d6c2c58bb54e0eafb365625ac33871 -> Peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Couldn't send request to peer 819f9e3412e54d44abb11d8b3ca877c2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 84738a6c78b24c1fb64013048763b18f. This is attempt 1: this message will repeat every 5th retry.
I20250114 02:54:29.877589 1992 ts_tablet_manager.cc:1918] T 5ac7708550954a209e2a8d217c3b39bd P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:29.878063 1992 log.cc:1198] T 5ac7708550954a209e2a8d217c3b39bd P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/5ac7708550954a209e2a8d217c3b39bd
I20250114 02:54:29.878819 1992 ts_tablet_manager.cc:1939] T 5ac7708550954a209e2a8d217c3b39bd P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:29.880797 1992 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 235) took 3823 ms. Trace:
I20250114 02:54:29.895609 1207 tablet_bootstrap.cc:492] T a52d6749ff834b62a8e20a2f45935240 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:29.900833 1965 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55068 (request call id 270) took 3321 ms. Trace:
I20250114 02:54:29.901314 1965 rpcz_store.cc:276] 0114 02:54:26.579447 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:26.580667 (+ 1220us) service_pool.cc:225] Handling call
0114 02:54:29.381363 (+2800696us) ts_tablet_manager.cc:1145] Deleting tablet f0bd9bd5a17d418d8082ed0ebd1bf4cc
0114 02:54:29.381403 (+ 40us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:29.520195 (+138792us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:29.769982 (+249787us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:29.900770 (+130788us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2800130}
I20250114 02:54:29.902709 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet f0bd9bd5a17d418d8082ed0ebd1bf4cc (table default.Table_dc087fb027f74c4a8bd5dede1142d97d [id=971cb3ca793e4945ad07e6be966e01f3]) successfully deleted
I20250114 02:54:29.903788 1965 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55068 (request call id 271) took 3302 ms. Trace:
I20250114 02:54:29.904150 1965 rpcz_store.cc:276] 0114 02:54:26.601249 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:26.601380 (+ 131us) service_pool.cc:225] Handling call
0114 02:54:29.903537 (+3302157us) ts_tablet_manager.cc:1145] Deleting tablet 321b9ad6d1ac4d1c80c7a01faf2015af
0114 02:54:29.903579 (+ 42us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:29.903736 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3301571}
W20250114 02:54:29.905462 456 catalog_manager.cc:4839] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet 321b9ad6d1ac4d1c80c7a01faf2015af because the tablet was not found. No further retry: Not found: Tablet not found: 321b9ad6d1ac4d1c80c7a01faf2015af
I20250114 02:54:29.906462 1965 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55068 (request call id 272) took 3256 ms. Trace:
I20250114 02:54:29.907126 1965 rpcz_store.cc:276] 0114 02:54:26.650407 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:26.650583 (+ 176us) service_pool.cc:225] Handling call
0114 02:54:29.906226 (+3255643us) ts_tablet_manager.cc:1145] Deleting tablet 0f1ea1608213492c9a216757896ff2a4
0114 02:54:29.906268 (+ 42us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:29.906413 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3254993}
W20250114 02:54:29.908370 456 catalog_manager.cc:4839] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet 0f1ea1608213492c9a216757896ff2a4 because the tablet was not found. No further retry: Not found: Tablet not found: 0f1ea1608213492c9a216757896ff2a4
I20250114 02:54:29.909667 1965 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55068 (request call id 273) took 3217 ms. Trace:
I20250114 02:54:29.910995 1965 rpcz_store.cc:276] 0114 02:54:26.692661 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:26.692798 (+ 137us) service_pool.cc:225] Handling call
0114 02:54:29.909319 (+3216521us) ts_tablet_manager.cc:1145] Deleting tablet 6d741bfd00784d2d830be3ebe50dc7c4
0114 02:54:29.909357 (+ 38us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:29.909609 (+ 252us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3215945}
W20250114 02:54:29.912175 456 catalog_manager.cc:4852] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet 6d741bfd00784d2d830be3ebe50dc7c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6d741bfd00784d2d830be3ebe50dc7c4 already in progress: creating tablet
I20250114 02:54:29.914355 2133 consensus_queue.cc:1035] T 6e858a8449444ef999b9bbef612249d3 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:29.916326 1992 rpcz_store.cc:276] 0114 02:54:26.057041 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:26.057134 (+ 93us) service_pool.cc:225] Handling call
0114 02:54:29.531621 (+3474487us) ts_tablet_manager.cc:1145] Deleting tablet 5ac7708550954a209e2a8d217c3b39bd
0114 02:54:29.531664 (+ 43us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:29.695137 (+163473us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:29.877377 (+182240us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:29.880726 (+ 3349us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3466236}
I20250114 02:54:29.917815 1992 tablet_replica.cc:331] T de4ca83a69704a368a25954cdc1b9392 P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:29.918556 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet 5ac7708550954a209e2a8d217c3b39bd (table default.Table_ee814dd266e64286864db0eac9e11664 [id=13151866bf2b4f44b1dd5c92380a4d43]) successfully deleted
W20250114 02:54:29.924533 464 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 141cede72b8741a8b0615d7c6bb823bb consensus state has no leader
I20250114 02:54:29.929155 1206 tablet_bootstrap.cc:492] T fc406eab968745d893f1a8538ff61f37 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:29.929641 1206 ts_tablet_manager.cc:1397] T fc406eab968745d893f1a8538ff61f37 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.341s user 0.005s sys 0.005s
I20250114 02:54:29.936076 2016 raft_consensus.cc:491] T a52d6749ff834b62a8e20a2f45935240 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:29.936559 2016 raft_consensus.cc:513] T a52d6749ff834b62a8e20a2f45935240 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:29.938279 2016 leader_election.cc:290] T a52d6749ff834b62a8e20a2f45935240 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:29.941373 1102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a52d6749ff834b62a8e20a2f45935240" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:29.942549 1990 ts_tablet_manager.cc:1918] T de4ca83a69704a368a25954cdc1b9392 P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
W20250114 02:54:29.942673 898 leader_election.cc:343] T a52d6749ff834b62a8e20a2f45935240 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:29.965953 811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a52d6749ff834b62a8e20a2f45935240" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:29.966912 811 raft_consensus.cc:2463] T a52d6749ff834b62a8e20a2f45935240 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:29.967417 935 tablet_service.cc:1467] Processing CreateTablet for tablet 6585ba7336374e248f9044038b9c340c (DEFAULT_TABLE table=default.Table_23e9951e382b44538252984608700dcc [id=804e11d5d5e64f2897ff2265ff77427d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:29.968567 935 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6585ba7336374e248f9044038b9c340c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:29.970331 2109 tablet.cc:1721] T 9bfd8fd6d2104fb4a6484cb8c98981bd P 819f9e3412e54d44abb11d8b3ca877c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250114 02:54:29.976941 467 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet edc69b66bea54625a4b15867b75986b7 (table default.Table_0a0bda55a4a9476c98d961822b8a19bd [id=fb2be5c1ca9444a197f3315ea9c59b09]). Expected version 1 got 0
I20250114 02:54:29.978255 467 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 38fe8829d98547fb907ef78099df3d7f (table default.Table_60450873174a4c80aa7e2122343d82be [id=9299683349354f879847b7c9d93f6072]). Expected version 1 got 0
I20250114 02:54:29.981758 2194 tablet.cc:1721] T 15704785f56e43a3af69f181c75877ce P 819f9e3412e54d44abb11d8b3ca877c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250114 02:54:29.988361 1205 ts_tablet_manager.cc:1428] T a52d6749ff834b62a8e20a2f45935240 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.204s user 0.003s sys 0.003s
I20250114 02:54:29.988178 934 tablet_service.cc:1467] Processing CreateTablet for tablet e1d281decf7e4c52b075d9c7a84a66d2 (DEFAULT_TABLE table=default.Table_174825b874fc410fba44e7a1335609d8 [id=bb1b35430a7b4fa5a8b65d9c68133093]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:29.989387 934 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1d281decf7e4c52b075d9c7a84a66d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:29.991716 786 tablet_service.cc:1467] Processing CreateTablet for tablet 6585ba7336374e248f9044038b9c340c (DEFAULT_TABLE table=default.Table_23e9951e382b44538252984608700dcc [id=804e11d5d5e64f2897ff2265ff77427d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:29.992785 786 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6585ba7336374e248f9044038b9c340c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:29.996855 794 tablet_service.cc:1467] Processing CreateTablet for tablet e1d281decf7e4c52b075d9c7a84a66d2 (DEFAULT_TABLE table=default.Table_174825b874fc410fba44e7a1335609d8 [id=bb1b35430a7b4fa5a8b65d9c68133093]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:29.997691 794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1d281decf7e4c52b075d9c7a84a66d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:29.997831 464 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_82cba210b70a491290d9ec065423f7c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:29.999382 464 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_82cba210b70a491290d9ec065423f7c1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250114 02:54:29.944054 1040 consensus_peers.cc:487] T 6e858a8449444ef999b9bbef612249d3 P 90d6c2c58bb54e0eafb365625ac33871 -> Peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Couldn't send request to peer 525198593c8e47afaee54e74267b87e3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250114 02:54:30.000447 1040 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.66:44037, user_credentials={real_user=slave}} blocked reactor thread for 56489.3us
I20250114 02:54:30.002177 1086 tablet_service.cc:1467] Processing CreateTablet for tablet 6585ba7336374e248f9044038b9c340c (DEFAULT_TABLE table=default.Table_23e9951e382b44538252984608700dcc [id=804e11d5d5e64f2897ff2265ff77427d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:30.003146 1086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6585ba7336374e248f9044038b9c340c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:30.003413 1075 tablet_service.cc:1467] Processing CreateTablet for tablet e1d281decf7e4c52b075d9c7a84a66d2 (DEFAULT_TABLE table=default.Table_174825b874fc410fba44e7a1335609d8 [id=bb1b35430a7b4fa5a8b65d9c68133093]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:30.004438 1075 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1d281decf7e4c52b075d9c7a84a66d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:30.012593 1990 log.cc:1198] T de4ca83a69704a368a25954cdc1b9392 P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/de4ca83a69704a368a25954cdc1b9392
I20250114 02:54:30.013382 1990 ts_tablet_manager.cc:1939] T de4ca83a69704a368a25954cdc1b9392 P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
I20250114 02:54:30.015211 1990 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 262) took 3384 ms. Trace:
I20250114 02:54:30.015571 1990 rpcz_store.cc:276] 0114 02:54:26.631247 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:26.713333 (+ 82086us) service_pool.cc:225] Handling call
0114 02:54:29.764320 (+3050987us) ts_tablet_manager.cc:1145] Deleting tablet de4ca83a69704a368a25954cdc1b9392
0114 02:54:29.764355 (+ 35us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:29.902344 (+137989us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:29.942335 (+ 39991us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:30.015158 (+ 72823us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2960302}
I20250114 02:54:30.019754 935 tablet_service.cc:1467] Processing CreateTablet for tablet 41814f82964a4bcb8dbb6d3738052448 (DEFAULT_TABLE table=default.Table_4ea70b8c32474894a62595dd9e0a1ae4 [id=1d695a9985a9462191e67e804f397768]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:30.020859 935 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41814f82964a4bcb8dbb6d3738052448. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:30.020673 787 tablet_service.cc:1467] Processing CreateTablet for tablet 41814f82964a4bcb8dbb6d3738052448 (DEFAULT_TABLE table=default.Table_4ea70b8c32474894a62595dd9e0a1ae4 [id=1d695a9985a9462191e67e804f397768]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:30.021723 787 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41814f82964a4bcb8dbb6d3738052448. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:29.931821 1206 raft_consensus.cc:357] T fc406eab968745d893f1a8538ff61f37 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:30.027125 1206 raft_consensus.cc:383] T fc406eab968745d893f1a8538ff61f37 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:30.027447 1206 raft_consensus.cc:738] T fc406eab968745d893f1a8538ff61f37 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:30.028193 1206 consensus_queue.cc:260] T fc406eab968745d893f1a8538ff61f37 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:30.029245 468 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_ada937b35c344a4b8d1c0a3d52fa23e2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:30.030839 468 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ada937b35c344a4b8d1c0a3d52fa23e2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:30.035005 1965 tablet_replica.cc:331] T 179248cb5acf4740b268e0ec8b2852f9 P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:30.035843 1965 raft_consensus.cc:2238] T 179248cb5acf4740b268e0ec8b2852f9 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:30.018489 1076 tablet_service.cc:1467] Processing CreateTablet for tablet 41814f82964a4bcb8dbb6d3738052448 (DEFAULT_TABLE table=default.Table_4ea70b8c32474894a62595dd9e0a1ae4 [id=1d695a9985a9462191e67e804f397768]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:30.036484 1965 raft_consensus.cc:2267] T 179248cb5acf4740b268e0ec8b2852f9 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:30.038478 1076 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41814f82964a4bcb8dbb6d3738052448. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:30.038511 1965 ts_tablet_manager.cc:1905] T 179248cb5acf4740b268e0ec8b2852f9 P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:30.040870 1992 raft_consensus.cc:2238] T de4ca83a69704a368a25954cdc1b9392 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:30.041425 1992 raft_consensus.cc:2267] T de4ca83a69704a368a25954cdc1b9392 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:30.043371 1992 ts_tablet_manager.cc:1905] T de4ca83a69704a368a25954cdc1b9392 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:30.044711 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet de4ca83a69704a368a25954cdc1b9392 (table default.Table_69b77a6fd8ad45cc9e3dc32b202d93cd [id=8536eb7f6bf542daa88a8db0e4873456]) successfully deleted
I20250114 02:54:30.044965 1990 tablet_replica.cc:331] T d3fb28717c6f4844af84bb817fe7dfc4 P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:30.045675 1990 raft_consensus.cc:2238] T d3fb28717c6f4844af84bb817fe7dfc4 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:30.046124 1990 raft_consensus.cc:2267] T d3fb28717c6f4844af84bb817fe7dfc4 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:30.047787 1990 ts_tablet_manager.cc:1905] T d3fb28717c6f4844af84bb817fe7dfc4 P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:30.047948 466 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_a573620245b049398a53d6cb8799ca39"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:30.050336 466 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a573620245b049398a53d6cb8799ca39 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:30.035239 2133 consensus_queue.cc:1035] T 84738a6c78b24c1fb64013048763b18f P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:30.053433 898 leader_election.cc:304] T a52d6749ff834b62a8e20a2f45935240 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters: 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:30.054690 2143 raft_consensus.cc:2798] T a52d6749ff834b62a8e20a2f45935240 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:30.055013 2143 raft_consensus.cc:491] T a52d6749ff834b62a8e20a2f45935240 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:30.055326 2143 raft_consensus.cc:3054] T a52d6749ff834b62a8e20a2f45935240 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:30.062829 2143 raft_consensus.cc:513] T a52d6749ff834b62a8e20a2f45935240 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:30.066277 1102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a52d6749ff834b62a8e20a2f45935240" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
W20250114 02:54:30.067526 898 leader_election.cc:343] T a52d6749ff834b62a8e20a2f45935240 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:30.069773 1205 tablet_bootstrap.cc:492] T d05089eae2b74f9fb23a226d358a59da P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:30.088374 2143 leader_election.cc:290] T a52d6749ff834b62a8e20a2f45935240 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Requested vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:30.091463 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a52d6749ff834b62a8e20a2f45935240" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:30.091995 818 raft_consensus.cc:3054] T a52d6749ff834b62a8e20a2f45935240 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:30.099784 935 tablet_service.cc:1467] Processing CreateTablet for tablet b8f78115ca2e4d70a710e6f965bdf66a (DEFAULT_TABLE table=default.Table_c62ee80bacab484cb74431dcb4723918 [id=01f1fa2a006c4bd4bc22e6648a2152dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:30.101863 818 raft_consensus.cc:2463] T a52d6749ff834b62a8e20a2f45935240 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:30.101967 1084 tablet_service.cc:1467] Processing CreateTablet for tablet 61e1a820fbd6444da12384d0def69828 (DEFAULT_TABLE table=default.Table_e23da46545d64974b1f1563734b3c21c [id=5a3e47eb050545e09755d7364d0a30fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:30.103030 1084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61e1a820fbd6444da12384d0def69828. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:30.103861 1075 tablet_service.cc:1467] Processing CreateTablet for tablet b8f78115ca2e4d70a710e6f965bdf66a (DEFAULT_TABLE table=default.Table_c62ee80bacab484cb74431dcb4723918 [id=01f1fa2a006c4bd4bc22e6648a2152dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:30.105105 1075 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8f78115ca2e4d70a710e6f965bdf66a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:30.121603 785 tablet_service.cc:1467] Processing CreateTablet for tablet b8f78115ca2e4d70a710e6f965bdf66a (DEFAULT_TABLE table=default.Table_c62ee80bacab484cb74431dcb4723918 [id=01f1fa2a006c4bd4bc22e6648a2152dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:30.143187 465 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet 4ab79e0a576843e89a2d524608c5b7f4
I20250114 02:54:30.145318 931 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ab79e0a576843e89a2d524608c5b7f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:29 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:30.147286 785 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8f78115ca2e4d70a710e6f965bdf66a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:30.147832 865 heartbeater.cc:346] Connected to a master server at 127.0.107.126:45367
I20250114 02:54:30.152453 2109 tablet.cc:1721] T 92cd8eccacf34ab7a66b3bbc69df3697 P 819f9e3412e54d44abb11d8b3ca877c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250114 02:54:30.156948 1076 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ab79e0a576843e89a2d524608c5b7f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:29 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:30.157613 469 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet 3161f75bc4e84000af022faa9804c810
I20250114 02:54:30.157815 2133 consensus_queue.cc:1035] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:30.158890 469 catalog_manager.cc:6596] ReplaceTablet: tablet 3161f75bc4e84000af022faa9804c810 deleted and replaced by tablet 53929328fa51408aa98d7825018b6fdb
I20250114 02:54:30.160902 931 tablet_service.cc:1514] Processing DeleteTablet for tablet 3161f75bc4e84000af022faa9804c810 with delete_type TABLET_DATA_DELETED (replaced by tablet 53929328fa51408aa98d7825018b6fdb) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:30.163028 1076 tablet_service.cc:1514] Processing DeleteTablet for tablet 3161f75bc4e84000af022faa9804c810 with delete_type TABLET_DATA_DELETED (replaced by tablet 53929328fa51408aa98d7825018b6fdb) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:30.185721 1206 ts_tablet_manager.cc:1428] T fc406eab968745d893f1a8538ff61f37 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.256s user 0.003s sys 0.002s
I20250114 02:54:30.185747 465 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_666fde8ba59a4b688d83a1a27adfb7f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:30.186796 1206 tablet_bootstrap.cc:492] T df045d80d8ce4ff194a6d2039a44bce6 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:30.188745 469 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_42771479a01b45f7ade52d2ec3896c4a" } new_table_name: "default.Table_57f2197ca94b407c9fac253e8abc311d" modify_external_catalogs: true
W20250114 02:54:30.187691 465 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_666fde8ba59a4b688d83a1a27adfb7f7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:30.192682 898 leader_election.cc:304] T a52d6749ff834b62a8e20a2f45935240 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters: 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:30.194130 2097 raft_consensus.cc:2798] T a52d6749ff834b62a8e20a2f45935240 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:30.194562 2097 raft_consensus.cc:695] T a52d6749ff834b62a8e20a2f45935240 P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Becoming Leader. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Running, Role: LEADER
I20250114 02:54:30.195271 2097 consensus_queue.cc:237] T a52d6749ff834b62a8e20a2f45935240 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:30.197211 784 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ab79e0a576843e89a2d524608c5b7f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:29 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:30.198242 784 tablet_service.cc:1514] Processing DeleteTablet for tablet 3161f75bc4e84000af022faa9804c810 with delete_type TABLET_DATA_DELETED (replaced by tablet 53929328fa51408aa98d7825018b6fdb) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:30.199199 2164 raft_consensus.cc:491] T ed7f575653c848d7b076c17c38c4a244 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:30.199656 2164 raft_consensus.cc:513] T ed7f575653c848d7b076c17c38c4a244 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:30.201584 2164 leader_election.cc:290] T ed7f575653c848d7b076c17c38c4a244 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:30.203073 959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7f575653c848d7b076c17c38c4a244" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
W20250114 02:54:30.204284 751 leader_election.cc:343] T ed7f575653c848d7b076c17c38c4a244 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:30.119797 467 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet a52d6749ff834b62a8e20a2f45935240 (table default.Table_1c9b7896941e4289b1c12e20c210c2c1 [id=b3ee15b9f10f4d489edbde3af9034d25]). Expected version 1 got 0
I20250114 02:54:30.208070 467 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 92cd8eccacf34ab7a66b3bbc69df3697 (table default.Table_990b8d5f1bd146d6b358457f954780c9 [id=26627ccdc808451fb9593ad7a6df6991]). Expected version 2 got 0
I20250114 02:54:30.208833 467 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 99b940d872ce4781907517c40fe08781 (table default.Table_0943b1ca347042c9a6043006bc566701 [id=e3418787ef0a4e22979fb41d4f94ebb0]). Expected version 1 got 0
I20250114 02:54:30.210003 467 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 719188f639724851a5fc2b37cb60fdbb (table default.Table_8f8a18d7868c44bbb4d66a0692b08186 [id=e6292728826f437cb75c524a8aef2e87]). Expected version 1 got 0
I20250114 02:54:30.211228 467 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 044832851f804208b0a371eb0857dbf3 (table default.Table_0518d8bed7304c86aff32e05e6f07c0d [id=515a2fe3f1e84e82bccc02ef5cecef1b]). Expected version 1 got 0
I20250114 02:54:30.225297 1990 ts_tablet_manager.cc:1918] T d3fb28717c6f4844af84bb817fe7dfc4 P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:30.225715 1990 log.cc:1198] T d3fb28717c6f4844af84bb817fe7dfc4 P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/d3fb28717c6f4844af84bb817fe7dfc4
I20250114 02:54:30.226389 1990 ts_tablet_manager.cc:1939] T d3fb28717c6f4844af84bb817fe7dfc4 P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
I20250114 02:54:30.233538 471 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet 25bce689024f49a49496e415efe91fdc
I20250114 02:54:30.235172 931 tablet_service.cc:1514] Processing DeleteTablet for tablet 25bce689024f49a49496e415efe91fdc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:29 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:30.086418 1207 tablet_bootstrap.cc:654] T a52d6749ff834b62a8e20a2f45935240 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:30.256430 931 tablet_service.cc:1467] Processing CreateTablet for tablet 61e1a820fbd6444da12384d0def69828 (DEFAULT_TABLE table=default.Table_e23da46545d64974b1f1563734b3c21c [id=5a3e47eb050545e09755d7364d0a30fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:30.256431 932 tablet_service.cc:1467] Processing CreateTablet for tablet cb8aa4212982417b861323c4809017d4 (DEFAULT_TABLE table=default.Table_1097f09fb10e441fb639f8d7aa12d487 [id=d8c9691818d042d7b49475fa009c4ca1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:30.257550 2016 consensus_queue.cc:1035] T edc69b66bea54625a4b15867b75986b7 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:30.259356 2016 consensus_queue.cc:1035] T 7774c2087422492696a58e0075921ae7 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:30.264262 1205 tablet_bootstrap.cc:700] T d05089eae2b74f9fb23a226d358a59da P 525198593c8e47afaee54e74267b87e3: Time spent opening tablet: real 0.190s user 0.003s sys 0.000s
I20250114 02:54:30.264854 1205 tablet_bootstrap.cc:654] T d05089eae2b74f9fb23a226d358a59da P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:30.279484 471 master_service.cc:946] ReplaceTablet: received request to replace tablet 99b940d872ce4781907517c40fe08781 from {username='slave'} at 127.0.0.1:33540
I20250114 02:54:30.306581 786 tablet_service.cc:1514] Processing DeleteTablet for tablet 25bce689024f49a49496e415efe91fdc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:29 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:30.312525 1075 tablet_service.cc:1514] Processing DeleteTablet for tablet 25bce689024f49a49496e415efe91fdc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:29 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:30.315510 794 tablet_service.cc:1467] Processing CreateTablet for tablet 61e1a820fbd6444da12384d0def69828 (DEFAULT_TABLE table=default.Table_e23da46545d64974b1f1563734b3c21c [id=5a3e47eb050545e09755d7364d0a30fe]), partition=RANGE (key) PARTITION UNBOUNDED
W20250114 02:54:30.319908 456 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 141cede72b8741a8b0615d7c6bb823bb (table default.Table_4364d1df16d149d2ba10469ce7288d4e [id=13fc996d911b430290e37a94dd44b1ab], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 141cede72b8741a8b0615d7c6bb823bb consensus state has no leader
I20250114 02:54:30.326596 935 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8f78115ca2e4d70a710e6f965bdf66a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:30.319092 784 tablet_service.cc:1467] Processing CreateTablet for tablet cb8aa4212982417b861323c4809017d4 (DEFAULT_TABLE table=default.Table_1097f09fb10e441fb639f8d7aa12d487 [id=d8c9691818d042d7b49475fa009c4ca1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:30.329546 784 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb8aa4212982417b861323c4809017d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:30.343462 931 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61e1a820fbd6444da12384d0def69828. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:30.348796 932 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb8aa4212982417b861323c4809017d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:30.374817 794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61e1a820fbd6444da12384d0def69828. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:30.389680 1205 tablet_bootstrap.cc:492] T d05089eae2b74f9fb23a226d358a59da P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:30.390183 1205 ts_tablet_manager.cc:1397] T d05089eae2b74f9fb23a226d358a59da P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.321s user 0.010s sys 0.000s
I20250114 02:54:30.400957 2143 consensus_queue.cc:1035] T edc69b66bea54625a4b15867b75986b7 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:30.288154 1206 tablet_bootstrap.cc:654] T df045d80d8ce4ff194a6d2039a44bce6 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:30.407493 1990 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 263) took 3693 ms. Trace:
I20250114 02:54:30.407881 1990 rpcz_store.cc:276] 0114 02:54:26.714130 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:26.714342 (+ 212us) service_pool.cc:225] Handling call
0114 02:54:30.044824 (+3330482us) ts_tablet_manager.cc:1145] Deleting tablet d3fb28717c6f4844af84bb817fe7dfc4
0114 02:54:30.044864 (+ 40us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:30.118183 (+ 73319us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:30.225044 (+106861us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:30.407424 (+182380us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13890,"tablet-delete.queue_time_us":3240641}
I20250114 02:54:30.408552 1102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7f575653c848d7b076c17c38c4a244" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:30.409559 1990 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 267) took 3617 ms. Trace:
I20250114 02:54:30.409924 1990 rpcz_store.cc:276] 0114 02:54:26.792526 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:26.823326 (+ 30800us) service_pool.cc:225] Handling call
0114 02:54:30.409173 (+3585847us) ts_tablet_manager.cc:1145] Deleting tablet 6d741bfd00784d2d830be3ebe50dc7c4
0114 02:54:30.409219 (+ 46us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:30.409503 (+ 284us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3585253}
I20250114 02:54:30.411021 1990 tablet_replica.cc:331] T 179248cb5acf4740b268e0ec8b2852f9 P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:30.412851 1084 tablet_service.cc:1467] Processing CreateTablet for tablet cb8aa4212982417b861323c4809017d4 (DEFAULT_TABLE table=default.Table_1097f09fb10e441fb639f8d7aa12d487 [id=d8c9691818d042d7b49475fa009c4ca1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:30.414852 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet d3fb28717c6f4844af84bb817fe7dfc4 (table default.Table_b8b1e6d9bd0742f7880a7f55e986d0ee [id=871e09c142b94b79aeff9bfd8d78e2bb]) successfully deleted
W20250114 02:54:30.416014 458 catalog_manager.cc:4852] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 6d741bfd00784d2d830be3ebe50dc7c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6d741bfd00784d2d830be3ebe50dc7c4 already in progress: creating tablet
W20250114 02:54:30.415781 753 leader_election.cc:343] T ed7f575653c848d7b076c17c38c4a244 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:30.418334 1990 raft_consensus.cc:2238] T 179248cb5acf4740b268e0ec8b2852f9 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:30.417454 753 leader_election.cc:304] T ed7f575653c848d7b076c17c38c4a244 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:30.419462 1820 consensus_queue.cc:1035] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250114 02:54:30.420431 753 consensus_peers.cc:487] T 179248cb5acf4740b268e0ec8b2852f9 P 819f9e3412e54d44abb11d8b3ca877c2 -> Peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Couldn't send request to peer 90d6c2c58bb54e0eafb365625ac33871. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20250114 02:54:30.420991 1990 raft_consensus.cc:2267] T 179248cb5acf4740b268e0ec8b2852f9 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:30.422966 1990 ts_tablet_manager.cc:1905] T 179248cb5acf4740b268e0ec8b2852f9 P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:30.424602 2143 consensus_queue.cc:1035] T 7774c2087422492696a58e0075921ae7 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:30.427757 1084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb8aa4212982417b861323c4809017d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:30.427799 1205 raft_consensus.cc:357] T d05089eae2b74f9fb23a226d358a59da P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:30.438844 1205 raft_consensus.cc:383] T d05089eae2b74f9fb23a226d358a59da P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:30.439172 1205 raft_consensus.cc:738] T d05089eae2b74f9fb23a226d358a59da P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:30.439980 1205 consensus_queue.cc:260] T d05089eae2b74f9fb23a226d358a59da P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:30.428400 2164 raft_consensus.cc:2743] T ed7f575653c848d7b076c17c38c4a244 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:30.455169 1205 ts_tablet_manager.cc:1428] T d05089eae2b74f9fb23a226d358a59da P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.065s user 0.005s sys 0.000s
I20250114 02:54:30.485165 1992 ts_tablet_manager.cc:1918] T de4ca83a69704a368a25954cdc1b9392 P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250114 02:54:30.485699 1992 log.cc:1198] T de4ca83a69704a368a25954cdc1b9392 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/de4ca83a69704a368a25954cdc1b9392
I20250114 02:54:30.486503 1992 ts_tablet_manager.cc:1939] T de4ca83a69704a368a25954cdc1b9392 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:30.488334 1992 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 239) took 3954 ms. Trace:
I20250114 02:54:30.488704 1992 rpcz_store.cc:276] 0114 02:54:26.533742 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:26.533880 (+ 138us) service_pool.cc:225] Handling call
0114 02:54:29.917634 (+3383754us) ts_tablet_manager.cc:1145] Deleting tablet de4ca83a69704a368a25954cdc1b9392
0114 02:54:29.917678 (+ 44us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:30.051795 (+134117us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:30.484915 (+433120us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:30.488278 (+ 3363us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":122039,"spinlock_wait_cycles":3328,"tablet-delete.queue_time_us":3383168}
I20250114 02:54:30.490301 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet de4ca83a69704a368a25954cdc1b9392 (table default.Table_69b77a6fd8ad45cc9e3dc32b202d93cd [id=8536eb7f6bf542daa88a8db0e4873456]) successfully deleted
I20250114 02:54:30.495582 1206 tablet_bootstrap.cc:492] T df045d80d8ce4ff194a6d2039a44bce6 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:30.495936 1992 tablet_replica.cc:331] T d3fb28717c6f4844af84bb817fe7dfc4 P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:30.497153 1992 raft_consensus.cc:2238] T d3fb28717c6f4844af84bb817fe7dfc4 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 LEADER]: Raft consensus shutting down.
I20250114 02:54:30.498024 1992 raft_consensus.cc:2267] T d3fb28717c6f4844af84bb817fe7dfc4 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:30.499783 1992 ts_tablet_manager.cc:1905] T d3fb28717c6f4844af84bb817fe7dfc4 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:30.495978 1206 ts_tablet_manager.cc:1397] T df045d80d8ce4ff194a6d2039a44bce6 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.309s user 0.010s sys 0.001s
I20250114 02:54:30.513029 1206 raft_consensus.cc:357] T df045d80d8ce4ff194a6d2039a44bce6 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:30.514046 1205 tablet_bootstrap.cc:492] T 6e858a8449444ef999b9bbef612249d3 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:30.514339 1206 raft_consensus.cc:383] T df045d80d8ce4ff194a6d2039a44bce6 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:30.516304 1206 raft_consensus.cc:738] T df045d80d8ce4ff194a6d2039a44bce6 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:30.517086 1206 consensus_queue.cc:260] T df045d80d8ce4ff194a6d2039a44bce6 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:30.519740 1205 tablet_bootstrap.cc:654] T 6e858a8449444ef999b9bbef612249d3 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
W20250114 02:54:30.535049 1040 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.66:44037, user_credentials={real_user=slave}} blocked reactor thread for 115291us
I20250114 02:54:30.535503 1206 ts_tablet_manager.cc:1428] T df045d80d8ce4ff194a6d2039a44bce6 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.031s user 0.005s sys 0.001s
I20250114 02:54:30.547008 1206 tablet_bootstrap.cc:492] T 97f4e061d3b94ee9982410b7ce135453 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:30.569043 2143 consensus_queue.cc:1035] T 38fe8829d98547fb907ef78099df3d7f P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:30.579003 1205 tablet_bootstrap.cc:492] T 6e858a8449444ef999b9bbef612249d3 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:30.579447 1205 ts_tablet_manager.cc:1397] T 6e858a8449444ef999b9bbef612249d3 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.066s user 0.009s sys 0.000s
I20250114 02:54:30.585585 1206 tablet_bootstrap.cc:654] T 97f4e061d3b94ee9982410b7ce135453 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:30.586136 1992 ts_tablet_manager.cc:1918] T d3fb28717c6f4844af84bb817fe7dfc4 P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:30.587757 1992 log.cc:1198] T d3fb28717c6f4844af84bb817fe7dfc4 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/d3fb28717c6f4844af84bb817fe7dfc4
I20250114 02:54:30.588563 1992 ts_tablet_manager.cc:1939] T d3fb28717c6f4844af84bb817fe7dfc4 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:30.589434 2151 raft_consensus.cc:491] T fc406eab968745d893f1a8538ff61f37 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:30.589854 2151 raft_consensus.cc:513] T fc406eab968745d893f1a8538ff61f37 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:30.591698 2151 leader_election.cc:290] T fc406eab968745d893f1a8538ff61f37 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:30.596012 1102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc406eab968745d893f1a8538ff61f37" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
W20250114 02:54:30.597751 753 leader_election.cc:343] T fc406eab968745d893f1a8538ff61f37 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:30.599252 957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc406eab968745d893f1a8538ff61f37" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
W20250114 02:54:30.603219 751 leader_election.cc:343] T fc406eab968745d893f1a8538ff61f37 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:30.603557 751 leader_election.cc:304] T fc406eab968745d893f1a8538ff61f37 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:30.604463 2151 raft_consensus.cc:2743] T fc406eab968745d893f1a8538ff61f37 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:30.607976 1992 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 240) took 4064 ms. Trace:
I20250114 02:54:30.609694 1992 rpcz_store.cc:276] 0114 02:54:26.543726 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:26.543840 (+ 114us) service_pool.cc:225] Handling call
0114 02:54:30.489981 (+3946141us) ts_tablet_manager.cc:1145] Deleting tablet d3fb28717c6f4844af84bb817fe7dfc4
0114 02:54:30.495746 (+ 5765us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:30.530053 (+ 34307us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:30.585932 (+ 55879us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:30.607920 (+ 21988us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3945541}
I20250114 02:54:30.610198 1205 raft_consensus.cc:357] T 6e858a8449444ef999b9bbef612249d3 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:30.611889 1992 tablet_replica.cc:331] T 179248cb5acf4740b268e0ec8b2852f9 P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:30.611958 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet d3fb28717c6f4844af84bb817fe7dfc4 (table default.Table_b8b1e6d9bd0742f7880a7f55e986d0ee [id=871e09c142b94b79aeff9bfd8d78e2bb]) successfully deleted
I20250114 02:54:30.612815 1992 raft_consensus.cc:2238] T 179248cb5acf4740b268e0ec8b2852f9 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 LEADER]: Raft consensus shutting down.
I20250114 02:54:30.613663 1992 raft_consensus.cc:2267] T 179248cb5acf4740b268e0ec8b2852f9 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:30.615494 1992 ts_tablet_manager.cc:1905] T 179248cb5acf4740b268e0ec8b2852f9 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:30.616629 1205 raft_consensus.cc:383] T 6e858a8449444ef999b9bbef612249d3 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:30.618156 1102 raft_consensus.cc:3054] T 38fe8829d98547fb907ef78099df3d7f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:30.619009 1205 raft_consensus.cc:738] T 6e858a8449444ef999b9bbef612249d3 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:30.619746 1965 ts_tablet_manager.cc:1918] T 179248cb5acf4740b268e0ec8b2852f9 P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250114 02:54:30.638499 1206 tablet_bootstrap.cc:492] T 97f4e061d3b94ee9982410b7ce135453 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:30.638921 1206 ts_tablet_manager.cc:1397] T 97f4e061d3b94ee9982410b7ce135453 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.092s user 0.007s sys 0.004s
W20250114 02:54:30.640220 455 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.124:39105, user_credentials={real_user=slave}} blocked reactor thread for 102283us
I20250114 02:54:30.642412 1206 raft_consensus.cc:357] T 97f4e061d3b94ee9982410b7ce135453 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:30.647523 1206 raft_consensus.cc:383] T 97f4e061d3b94ee9982410b7ce135453 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:30.647511 2201 raft_consensus.cc:491] T 38fe8829d98547fb907ef78099df3d7f P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:30.647835 1206 raft_consensus.cc:738] T 97f4e061d3b94ee9982410b7ce135453 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:30.647961 2201 raft_consensus.cc:513] T 38fe8829d98547fb907ef78099df3d7f P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:30.648779 1206 consensus_queue.cc:260] T 97f4e061d3b94ee9982410b7ce135453 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:30.649879 2201 leader_election.cc:290] T 38fe8829d98547fb907ef78099df3d7f P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:30.652771 813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38fe8829d98547fb907ef78099df3d7f" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:30.654671 1206 ts_tablet_manager.cc:1428] T 97f4e061d3b94ee9982410b7ce135453 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.015s user 0.004s sys 0.001s
I20250114 02:54:30.657516 1207 tablet_bootstrap.cc:492] T a52d6749ff834b62a8e20a2f45935240 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:30.657994 1207 ts_tablet_manager.cc:1397] T a52d6749ff834b62a8e20a2f45935240 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.763s user 0.013s sys 0.005s
I20250114 02:54:30.661111 1206 tablet_bootstrap.cc:492] T 3161f75bc4e84000af022faa9804c810 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:30.666169 1990 ts_tablet_manager.cc:1918] T 179248cb5acf4740b268e0ec8b2852f9 P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250114 02:54:30.666381 1206 tablet_bootstrap.cc:654] T 3161f75bc4e84000af022faa9804c810 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:30.666667 1990 log.cc:1198] T 179248cb5acf4740b268e0ec8b2852f9 P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/179248cb5acf4740b268e0ec8b2852f9
I20250114 02:54:30.667639 1990 ts_tablet_manager.cc:1939] T 179248cb5acf4740b268e0ec8b2852f9 P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
I20250114 02:54:30.671187 1207 raft_consensus.cc:357] T a52d6749ff834b62a8e20a2f45935240 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:30.672003 1207 raft_consensus.cc:383] T a52d6749ff834b62a8e20a2f45935240 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:30.672611 1207 raft_consensus.cc:738] T a52d6749ff834b62a8e20a2f45935240 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:30.673439 1207 consensus_queue.cc:260] T a52d6749ff834b62a8e20a2f45935240 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:30.674345 1992 ts_tablet_manager.cc:1918] T 179248cb5acf4740b268e0ec8b2852f9 P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20250114 02:54:30.680647 455 consensus_peers.cc:487] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750 -> Peer 87b3db09ac87436e9a60d6de5fedaff5 (127.0.107.124:39105): Couldn't send request to peer 87b3db09ac87436e9a60d6de5fedaff5. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.124:39105: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20250114 02:54:30.684392 1992 log.cc:1198] T 179248cb5acf4740b268e0ec8b2852f9 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/179248cb5acf4740b268e0ec8b2852f9
I20250114 02:54:30.685173 1992 ts_tablet_manager.cc:1939] T 179248cb5acf4740b268e0ec8b2852f9 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:30.695154 1992 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 242) took 3853 ms. Trace:
I20250114 02:54:30.697248 1992 rpcz_store.cc:276] 0114 02:54:26.841329 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:26.841472 (+ 143us) service_pool.cc:225] Handling call
0114 02:54:30.611752 (+3770280us) ts_tablet_manager.cc:1145] Deleting tablet 179248cb5acf4740b268e0ec8b2852f9
0114 02:54:30.611793 (+ 41us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:30.667694 (+ 55901us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:30.674152 (+ 6458us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:30.695094 (+ 20942us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3769765}
I20250114 02:54:30.620157 1205 consensus_queue.cc:260] T 6e858a8449444ef999b9bbef612249d3 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:30.700250 1992 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 241) took 4006 ms. Trace:
I20250114 02:54:30.700660 1205 ts_tablet_manager.cc:1428] T 6e858a8449444ef999b9bbef612249d3 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.121s user 0.006s sys 0.000s
I20250114 02:54:30.700531 1992 rpcz_store.cc:276] 0114 02:54:26.693993 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:26.864608 (+170615us) service_pool.cc:225] Handling call
0114 02:54:30.699872 (+3835264us) ts_tablet_manager.cc:1145] Deleting tablet 6d741bfd00784d2d830be3ebe50dc7c4
0114 02:54:30.699912 (+ 40us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:30.700183 (+ 271us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3834615}
I20250114 02:54:30.627454 2143 consensus_queue.cc:1035] T 38fe8829d98547fb907ef78099df3d7f P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:30.651907 957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38fe8829d98547fb907ef78099df3d7f" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:30.702325 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet 179248cb5acf4740b268e0ec8b2852f9 (table default.Table_58e014a2cfee480184ff3c66b1768cf6 [id=5745501ecd7b4cecaca3d35502a6cc40]) successfully deleted
I20250114 02:54:30.702877 1206 tablet_bootstrap.cc:492] T 3161f75bc4e84000af022faa9804c810 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
W20250114 02:54:30.702953 458 catalog_manager.cc:4852] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet 6d741bfd00784d2d830be3ebe50dc7c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6d741bfd00784d2d830be3ebe50dc7c4 already in progress: creating tablet
I20250114 02:54:30.703248 1206 ts_tablet_manager.cc:1397] T 3161f75bc4e84000af022faa9804c810 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.042s user 0.009s sys 0.001s
I20250114 02:54:30.705364 1206 raft_consensus.cc:357] T 3161f75bc4e84000af022faa9804c810 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:30.705969 1206 raft_consensus.cc:383] T 3161f75bc4e84000af022faa9804c810 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:30.706609 1206 raft_consensus.cc:738] T 3161f75bc4e84000af022faa9804c810 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:30.707520 1992 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 243) took 3522 ms. Trace:
I20250114 02:54:30.708431 1205 tablet_bootstrap.cc:492] T 6e2fb05f72b743e2a02ed36b63f80b23 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:30.708880 1992 rpcz_store.cc:276] 0114 02:54:27.185064 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:27.185457 (+ 393us) service_pool.cc:225] Handling call
0114 02:54:30.707207 (+3521750us) ts_tablet_manager.cc:1145] Deleting tablet a2c3f3a89b3d47f1b0dfacfac8218805
0114 02:54:30.707243 (+ 36us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:30.707464 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3521240}
I20250114 02:54:30.708195 1206 consensus_queue.cc:260] T 3161f75bc4e84000af022faa9804c810 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
W20250114 02:54:30.709893 458 catalog_manager.cc:4852] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet a2c3f3a89b3d47f1b0dfacfac8218805 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2c3f3a89b3d47f1b0dfacfac8218805 already in progress: creating tablet
I20250114 02:54:30.714267 2151 raft_consensus.cc:491] T 97f4e061d3b94ee9982410b7ce135453 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:30.714641 2151 raft_consensus.cc:513] T 97f4e061d3b94ee9982410b7ce135453 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:30.717397 957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97f4e061d3b94ee9982410b7ce135453" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:30.717661 1206 ts_tablet_manager.cc:1428] T 3161f75bc4e84000af022faa9804c810 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20250114 02:54:30.717660 1205 tablet_bootstrap.cc:654] T 6e2fb05f72b743e2a02ed36b63f80b23 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
W20250114 02:54:30.718784 751 leader_election.cc:343] T 97f4e061d3b94ee9982410b7ce135453 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:30.719624 2151 leader_election.cc:290] T 97f4e061d3b94ee9982410b7ce135453 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:30.720983 1965 log.cc:1198] T 179248cb5acf4740b268e0ec8b2852f9 P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/179248cb5acf4740b268e0ec8b2852f9
I20250114 02:54:30.722900 1965 ts_tablet_manager.cc:1939] T 179248cb5acf4740b268e0ec8b2852f9 P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:30.725265 1965 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55068 (request call id 274) took 3885 ms. Trace:
I20250114 02:54:30.731739 1206 tablet_bootstrap.cc:492] T 141cede72b8741a8b0615d7c6bb823bb P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
W20250114 02:54:30.731325 455 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.125:46837, user_credentials={real_user=slave}} blocked reactor thread for 49956.1us
I20250114 02:54:30.747764 1992 tablet_replica.cc:331] T ed7f575653c848d7b076c17c38c4a244 P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:30.753113 1206 tablet_bootstrap.cc:654] T 141cede72b8741a8b0615d7c6bb823bb P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:30.759971 1992 raft_consensus.cc:2238] T ed7f575653c848d7b076c17c38c4a244 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:30.794034 1992 raft_consensus.cc:2267] T ed7f575653c848d7b076c17c38c4a244 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:30.795771 1992 ts_tablet_manager.cc:1905] T ed7f575653c848d7b076c17c38c4a244 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:30.702642 1040 leader_election.cc:304] T 38fe8829d98547fb907ef78099df3d7f P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90d6c2c58bb54e0eafb365625ac33871; no voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2
W20250114 02:54:30.800801 1040 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.66:44037, user_credentials={real_user=slave}} blocked reactor thread for 98235.6us
I20250114 02:54:30.802579 2201 raft_consensus.cc:2743] T 38fe8829d98547fb907ef78099df3d7f P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:54:30.804101 1102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97f4e061d3b94ee9982410b7ce135453" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
W20250114 02:54:30.807410 753 leader_election.cc:343] T 97f4e061d3b94ee9982410b7ce135453 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:30.807821 753 leader_election.cc:304] T 97f4e061d3b94ee9982410b7ce135453 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:30.808754 2151 raft_consensus.cc:2743] T 97f4e061d3b94ee9982410b7ce135453 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:30.815748 1206 tablet_bootstrap.cc:492] T 141cede72b8741a8b0615d7c6bb823bb P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:30.816146 1206 ts_tablet_manager.cc:1397] T 141cede72b8741a8b0615d7c6bb823bb P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.085s user 0.006s sys 0.003s
I20250114 02:54:30.819599 1990 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 268) took 3668 ms. Trace:
I20250114 02:54:30.820001 1990 rpcz_store.cc:276] 0114 02:54:27.151155 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:27.151288 (+ 133us) service_pool.cc:225] Handling call
0114 02:54:30.410853 (+3259565us) ts_tablet_manager.cc:1145] Deleting tablet 179248cb5acf4740b268e0ec8b2852f9
0114 02:54:30.410899 (+ 46us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:30.535486 (+124587us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:30.665949 (+130463us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:30.819539 (+153590us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3259019}
I20250114 02:54:30.828302 1965 rpcz_store.cc:276] 0114 02:54:26.840023 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:26.840165 (+ 142us) service_pool.cc:225] Handling call
0114 02:54:30.033980 (+3193815us) ts_tablet_manager.cc:1145] Deleting tablet 179248cb5acf4740b268e0ec8b2852f9
0114 02:54:30.034886 (+ 906us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:30.327936 (+293050us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:30.619523 (+291587us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:30.725205 (+105682us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3178023}
I20250114 02:54:30.829586 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet 179248cb5acf4740b268e0ec8b2852f9 (table default.Table_58e014a2cfee480184ff3c66b1768cf6 [id=5745501ecd7b4cecaca3d35502a6cc40]) successfully deleted
I20250114 02:54:30.818397 1206 raft_consensus.cc:357] T 141cede72b8741a8b0615d7c6bb823bb P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:30.836606 1206 raft_consensus.cc:383] T 141cede72b8741a8b0615d7c6bb823bb P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:30.836916 1206 raft_consensus.cc:738] T 141cede72b8741a8b0615d7c6bb823bb P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:30.837651 1206 consensus_queue.cc:260] T 141cede72b8741a8b0615d7c6bb823bb P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:30.844991 1205 tablet_bootstrap.cc:492] T 6e2fb05f72b743e2a02ed36b63f80b23 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:30.845417 1205 ts_tablet_manager.cc:1397] T 6e2fb05f72b743e2a02ed36b63f80b23 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.138s user 0.010s sys 0.000s
I20250114 02:54:30.847631 1205 raft_consensus.cc:357] T 6e2fb05f72b743e2a02ed36b63f80b23 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:30.848356 1205 raft_consensus.cc:383] T 6e2fb05f72b743e2a02ed36b63f80b23 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:30.848721 1205 raft_consensus.cc:738] T 6e2fb05f72b743e2a02ed36b63f80b23 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:30.825485 1207 ts_tablet_manager.cc:1428] T a52d6749ff834b62a8e20a2f45935240 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.167s user 0.006s sys 0.000s
I20250114 02:54:30.849454 1205 consensus_queue.cc:260] T 6e2fb05f72b743e2a02ed36b63f80b23 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:30.857265 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet 179248cb5acf4740b268e0ec8b2852f9 (table default.Table_58e014a2cfee480184ff3c66b1768cf6 [id=5745501ecd7b4cecaca3d35502a6cc40]) successfully deleted
I20250114 02:54:30.862915 1965 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55068 (request call id 275) took 3679 ms. Trace:
I20250114 02:54:30.863299 1965 rpcz_store.cc:276] 0114 02:54:27.183133 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:27.183284 (+ 151us) service_pool.cc:225] Handling call
0114 02:54:30.829641 (+3646357us) ts_tablet_manager.cc:1145] Deleting tablet a2c3f3a89b3d47f1b0dfacfac8218805
0114 02:54:30.862585 (+ 32944us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:30.862848 (+ 263us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3619257}
W20250114 02:54:30.864435 456 catalog_manager.cc:4852] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet a2c3f3a89b3d47f1b0dfacfac8218805 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2c3f3a89b3d47f1b0dfacfac8218805 already in progress: creating tablet
I20250114 02:54:30.865549 1205 ts_tablet_manager.cc:1428] T 6e2fb05f72b743e2a02ed36b63f80b23 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.020s user 0.005s sys 0.000s
I20250114 02:54:30.866850 1206 ts_tablet_manager.cc:1428] T 141cede72b8741a8b0615d7c6bb823bb P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.050s user 0.005s sys 0.001s
I20250114 02:54:30.867255 959 raft_consensus.cc:3054] T d05089eae2b74f9fb23a226d358a59da P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:30.872400 1206 tablet_bootstrap.cc:492] T f40c66314342453b898da3ac0e7e86bf P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:30.872637 1990 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 269) took 3483 ms. Trace:
I20250114 02:54:30.872964 1990 rpcz_store.cc:276] 0114 02:54:27.389502 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:27.390041 (+ 539us) service_pool.cc:225] Handling call
0114 02:54:30.858139 (+3468098us) ts_tablet_manager.cc:1145] Deleting tablet a2c3f3a89b3d47f1b0dfacfac8218805
0114 02:54:30.872290 (+ 14151us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:30.872587 (+ 297us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3467512}
I20250114 02:54:30.874082 1990 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 270) took 3435 ms. Trace:
I20250114 02:54:30.874574 1990 rpcz_store.cc:276] 0114 02:54:27.438221 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:27.438335 (+ 114us) service_pool.cc:225] Handling call
0114 02:54:30.873768 (+3435433us) ts_tablet_manager.cc:1145] Deleting tablet ed7f575653c848d7b076c17c38c4a244
0114 02:54:30.873806 (+ 38us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:30.874033 (+ 227us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3434829}
I20250114 02:54:30.875792 1990 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 274) took 2883 ms. Trace:
W20250114 02:54:30.876730 458 catalog_manager.cc:4852] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet a2c3f3a89b3d47f1b0dfacfac8218805 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2c3f3a89b3d47f1b0dfacfac8218805 already in progress: creating tablet
W20250114 02:54:30.879523 458 catalog_manager.cc:4852] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet ed7f575653c848d7b076c17c38c4a244 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed7f575653c848d7b076c17c38c4a244 already in progress: creating tablet
I20250114 02:54:30.882561 2144 consensus_queue.cc:1035] T d05089eae2b74f9fb23a226d358a59da P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:30.861835 1207 tablet_bootstrap.cc:492] T c86562688d5b4fffb71cbdf7be0afc37 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:30.897787 1965 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55068 (request call id 276) took 3680 ms. Trace:
I20250114 02:54:30.898118 1965 rpcz_store.cc:276] 0114 02:54:27.217329 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:27.217458 (+ 129us) service_pool.cc:225] Handling call
0114 02:54:30.864459 (+3647001us) ts_tablet_manager.cc:1145] Deleting tablet ed7f575653c848d7b076c17c38c4a244
0114 02:54:30.897489 (+ 33030us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:30.897734 (+ 245us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3646434}
I20250114 02:54:30.899194 1965 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55068 (request call id 281) took 3302 ms. Trace:
I20250114 02:54:30.899478 1965 rpcz_store.cc:276] 0114 02:54:27.596694 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:27.596815 (+ 121us) service_pool.cc:225] Handling call
0114 02:54:30.898889 (+3302074us) ts_tablet_manager.cc:1145] Deleting tablet 0b8361cd538046e0bcb4aac821748f67
0114 02:54:30.898926 (+ 37us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:30.899146 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3216431}
W20250114 02:54:30.900884 456 catalog_manager.cc:4852] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet ed7f575653c848d7b076c17c38c4a244 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed7f575653c848d7b076c17c38c4a244 already in progress: creating tablet
I20250114 02:54:30.900835 465 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_bacb81fcfeea434abb622c33872e1fde" } modify_external_catalogs: true
I20250114 02:54:30.901433 465 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_bacb81fcfeea434abb622c33872e1fde" } modify_external_catalogs: true
W20250114 02:54:30.901611 456 catalog_manager.cc:4852] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet 0b8361cd538046e0bcb4aac821748f67 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0b8361cd538046e0bcb4aac821748f67 already in progress: creating tablet
I20250114 02:54:30.904987 1206 tablet_bootstrap.cc:654] T f40c66314342453b898da3ac0e7e86bf P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
W20250114 02:54:30.905269 469 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5fa22baf436f4815a136673b12a476c7 consensus state has no leader
I20250114 02:54:30.906170 469 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_4ea70b8c32474894a62595dd9e0a1ae4" } modify_external_catalogs: true
I20250114 02:54:30.906512 1207 tablet_bootstrap.cc:654] T c86562688d5b4fffb71cbdf7be0afc37 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:30.906658 469 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_4ea70b8c32474894a62595dd9e0a1ae4" } modify_external_catalogs: true
I20250114 02:54:30.908814 471 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet 99b940d872ce4781907517c40fe08781
I20250114 02:54:30.909925 471 catalog_manager.cc:6596] ReplaceTablet: tablet 99b940d872ce4781907517c40fe08781 deleted and replaced by tablet fbeddf984d1f466192f60033715855ad
I20250114 02:54:30.910548 471 master_service.cc:946] ReplaceTablet: received request to replace tablet 232d6a697f744e179a1a8da492a7d227 from {username='slave'} at 127.0.0.1:33540
I20250114 02:54:30.908747 1423 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 02:54:30.876121 1990 rpcz_store.cc:276] 0114 02:54:27.992848 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:27.996292 (+ 3444us) service_pool.cc:225] Handling call
0114 02:54:30.875322 (+2879030us) ts_tablet_manager.cc:1145] Deleting tablet 0b8361cd538046e0bcb4aac821748f67
0114 02:54:30.875355 (+ 33us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:30.875745 (+ 390us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2878515}
I20250114 02:54:30.913789 1423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 02:54:30.914361 1423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 02:54:30.919682 1990 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 275) took 2926 ms. Trace:
W20250114 02:54:30.919821 458 catalog_manager.cc:4852] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 0b8361cd538046e0bcb4aac821748f67 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0b8361cd538046e0bcb4aac821748f67 already in progress: creating tablet
I20250114 02:54:30.919903 794 tablet_service.cc:1514] Processing DeleteTablet for tablet 99b940d872ce4781907517c40fe08781 with delete_type TABLET_DATA_DELETED (replaced by tablet fbeddf984d1f466192f60033715855ad) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:30.921270 1990 rpcz_store.cc:276] 0114 02:54:27.993099 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:27.997290 (+ 4191us) service_pool.cc:225] Handling call
0114 02:54:30.919326 (+2922036us) ts_tablet_manager.cc:1145] Deleting tablet ddad7584337345109ce102702008ab37
0114 02:54:30.919368 (+ 42us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:30.919628 (+ 260us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2921457}
I20250114 02:54:30.924211 1990 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 282) took 2894 ms. Trace:
I20250114 02:54:30.928151 463 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 15704785f56e43a3af69f181c75877ce (table default.Table_9da872c9a22e4d07947df8db0524e61f [id=1f964e9ed159449580fe5326673c18ab]). Expected version 1 got 0
I20250114 02:54:30.929172 463 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 9bfd8fd6d2104fb4a6484cb8c98981bd (table default.Table_0139fab1a81440309d9d7729c3e76c8a [id=2688bb1e2f4f4765986f738256d17f78]). Expected version 1 got 0
I20250114 02:54:30.929483 1206 tablet_bootstrap.cc:492] T f40c66314342453b898da3ac0e7e86bf P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:30.929687 463 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 38fe8829d98547fb907ef78099df3d7f (table default.Table_60450873174a4c80aa7e2122343d82be [id=9299683349354f879847b7c9d93f6072]). Expected version 1 got 0
I20250114 02:54:30.929883 1206 ts_tablet_manager.cc:1397] T f40c66314342453b898da3ac0e7e86bf P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.058s user 0.009s sys 0.000s
I20250114 02:54:30.930218 463 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet edc69b66bea54625a4b15867b75986b7 (table default.Table_0a0bda55a4a9476c98d961822b8a19bd [id=fb2be5c1ca9444a197f3315ea9c59b09]). Expected version 1 got 0
I20250114 02:54:30.928779 1084 tablet_service.cc:1514] Processing DeleteTablet for tablet 99b940d872ce4781907517c40fe08781 with delete_type TABLET_DATA_DELETED (replaced by tablet fbeddf984d1f466192f60033715855ad) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:30.930909 1990 rpcz_store.cc:276] 0114 02:54:28.029817 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:28.029978 (+ 161us) service_pool.cc:225] Handling call
0114 02:54:30.923870 (+2893892us) ts_tablet_manager.cc:1145] Deleting tablet d56255ca806d4e2cac09fbb5e1b1efbd
0114 02:54:30.923907 (+ 37us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:30.924160 (+ 253us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2893361}
I20250114 02:54:30.931375 463 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 719188f639724851a5fc2b37cb60fdbb (table default.Table_8f8a18d7868c44bbb4d66a0692b08186 [id=e6292728826f437cb75c524a8aef2e87]). Expected version 1 got 0
I20250114 02:54:30.931993 1206 raft_consensus.cc:357] T f40c66314342453b898da3ac0e7e86bf P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:30.932722 1206 raft_consensus.cc:383] T f40c66314342453b898da3ac0e7e86bf P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:30.932968 1206 raft_consensus.cc:738] T f40c66314342453b898da3ac0e7e86bf P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
W20250114 02:54:30.926986 458 catalog_manager.cc:4852] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet ddad7584337345109ce102702008ab37 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ddad7584337345109ce102702008ab37 already in progress: creating tablet
W20250114 02:54:30.934530 458 catalog_manager.cc:4852] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet d56255ca806d4e2cac09fbb5e1b1efbd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d56255ca806d4e2cac09fbb5e1b1efbd already in progress: creating tablet
I20250114 02:54:30.935942 1206 consensus_queue.cc:260] T f40c66314342453b898da3ac0e7e86bf P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:30.936827 1990 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 283) took 2873 ms. Trace:
I20250114 02:54:30.937120 1990 rpcz_store.cc:276] 0114 02:54:28.063058 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:28.084661 (+ 21603us) service_pool.cc:225] Handling call
0114 02:54:30.936516 (+2851855us) ts_tablet_manager.cc:1145] Deleting tablet 992bb16197b74be5b1dc884f29d9c240
0114 02:54:30.936554 (+ 38us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:30.936778 (+ 224us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2850914}
W20250114 02:54:30.938253 458 catalog_manager.cc:4852] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 992bb16197b74be5b1dc884f29d9c240 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 992bb16197b74be5b1dc884f29d9c240 already in progress: creating tablet
I20250114 02:54:30.944783 2201 consensus_queue.cc:1035] T d05089eae2b74f9fb23a226d358a59da P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:30.949573 1990 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 293) took 1975 ms. Trace:
I20250114 02:54:30.952181 1990 rpcz_store.cc:276] 0114 02:54:28.973897 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:28.975155 (+ 1258us) service_pool.cc:225] Handling call
0114 02:54:30.949248 (+1974093us) ts_tablet_manager.cc:1145] Deleting tablet 59a46bc865cc48df8e98598764fd1b54
0114 02:54:30.949288 (+ 40us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:30.949525 (+ 237us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1973581}
I20250114 02:54:30.958900 1965 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55068 (request call id 282) took 3352 ms. Trace:
I20250114 02:54:30.959316 1965 rpcz_store.cc:276] 0114 02:54:27.606513 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:27.606684 (+ 171us) service_pool.cc:225] Handling call
0114 02:54:30.900208 (+3293524us) ts_tablet_manager.cc:1145] Deleting tablet ddad7584337345109ce102702008ab37
0114 02:54:30.902058 (+ 1850us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:30.958823 (+ 56765us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3203779}
W20250114 02:54:30.960089 458 catalog_manager.cc:4852] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 59a46bc865cc48df8e98598764fd1b54 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59a46bc865cc48df8e98598764fd1b54 already in progress: creating tablet
I20250114 02:54:30.960850 1205 tablet_bootstrap.cc:492] T ed7f575653c848d7b076c17c38c4a244 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:30.961364 1990 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 301) took 1944 ms. Trace:
W20250114 02:54:30.961607 456 catalog_manager.cc:4852] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet ddad7584337345109ce102702008ab37 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ddad7584337345109ce102702008ab37 already in progress: creating tablet
I20250114 02:54:30.961699 1990 rpcz_store.cc:276] 0114 02:54:29.016958 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:29.022932 (+ 5974us) service_pool.cc:225] Handling call
0114 02:54:30.960973 (+1938041us) ts_tablet_manager.cc:1145] Deleting tablet 9ffec1d1bb594fda83df8d94d4533d56
0114 02:54:30.961011 (+ 38us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:30.961309 (+ 298us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1937311}
I20250114 02:54:30.963024 1990 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 302) took 1603 ms. Trace:
W20250114 02:54:30.963276 458 catalog_manager.cc:4852] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 9ffec1d1bb594fda83df8d94d4533d56 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9ffec1d1bb594fda83df8d94d4533d56 already in progress: creating tablet
I20250114 02:54:30.963681 1207 tablet_bootstrap.cc:492] T c86562688d5b4fffb71cbdf7be0afc37 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:30.963861 1206 ts_tablet_manager.cc:1428] T f40c66314342453b898da3ac0e7e86bf P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.034s user 0.007s sys 0.000s
I20250114 02:54:30.960633 931 tablet_service.cc:1514] Processing DeleteTablet for tablet 99b940d872ce4781907517c40fe08781 with delete_type TABLET_DATA_DELETED (replaced by tablet fbeddf984d1f466192f60033715855ad) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:30.963372 1990 rpcz_store.cc:276] 0114 02:54:29.364806 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:29.364945 (+ 139us) service_pool.cc:225] Handling call
0114 02:54:30.962795 (+1597850us) ts_tablet_manager.cc:1145] Deleting tablet bc46815b4803418c95ea9c039a4fae88
0114 02:54:30.962836 (+ 41us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:30.962979 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1597001}
I20250114 02:54:30.964088 1207 ts_tablet_manager.cc:1397] T c86562688d5b4fffb71cbdf7be0afc37 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.102s user 0.008s sys 0.000s
W20250114 02:54:30.965651 458 catalog_manager.cc:4839] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet bc46815b4803418c95ea9c039a4fae88 because the tablet was not found. No further retry: Not found: Tablet not found: bc46815b4803418c95ea9c039a4fae88
I20250114 02:54:30.971458 1965 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55068 (request call id 288) took 3050 ms. Trace:
I20250114 02:54:30.971881 1965 rpcz_store.cc:276] 0114 02:54:27.921154 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:27.921454 (+ 300us) service_pool.cc:225] Handling call
0114 02:54:30.960268 (+3038814us) ts_tablet_manager.cc:1145] Deleting tablet d56255ca806d4e2cac09fbb5e1b1efbd
0114 02:54:30.960318 (+ 50us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:30.971389 (+ 11071us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3038218}
I20250114 02:54:30.972733 1990 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 304) took 1604 ms. Trace:
I20250114 02:54:30.973242 1965 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55068 (request call id 289) took 2980 ms. Trace:
I20250114 02:54:30.973043 1990 rpcz_store.cc:276] 0114 02:54:29.368595 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:29.368693 (+ 98us) service_pool.cc:225] Handling call
0114 02:54:30.972489 (+1603796us) ts_tablet_manager.cc:1145] Deleting tablet ca986f81ed6848a89588366f3f400eec
0114 02:54:30.972529 (+ 40us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:30.972683 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1603160}
I20250114 02:54:30.973590 1965 rpcz_store.cc:276] 0114 02:54:27.992895 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:28.114813 (+121918us) service_pool.cc:225] Handling call
0114 02:54:30.972866 (+2858053us) ts_tablet_manager.cc:1145] Deleting tablet 992bb16197b74be5b1dc884f29d9c240
0114 02:54:30.972916 (+ 50us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:30.973185 (+ 269us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2857465}
W20250114 02:54:30.974090 458 catalog_manager.cc:4839] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet ca986f81ed6848a89588366f3f400eec because the tablet was not found. No further retry: Not found: Tablet not found: ca986f81ed6848a89588366f3f400eec
W20250114 02:54:30.975359 456 catalog_manager.cc:4852] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet d56255ca806d4e2cac09fbb5e1b1efbd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d56255ca806d4e2cac09fbb5e1b1efbd already in progress: creating tablet
W20250114 02:54:30.975976 456 catalog_manager.cc:4852] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet 992bb16197b74be5b1dc884f29d9c240 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 992bb16197b74be5b1dc884f29d9c240 already in progress: creating tablet
I20250114 02:54:30.975739 1207 raft_consensus.cc:357] T c86562688d5b4fffb71cbdf7be0afc37 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:30.979457 1207 raft_consensus.cc:383] T c86562688d5b4fffb71cbdf7be0afc37 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:30.980765 1207 raft_consensus.cc:738] T c86562688d5b4fffb71cbdf7be0afc37 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:30.982257 1206 tablet_bootstrap.cc:492] T 30a56fb2d941470590e17f37b90aef26 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:30.983461 1207 consensus_queue.cc:260] T c86562688d5b4fffb71cbdf7be0afc37 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:30.989575 1965 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55068 (request call id 299) took 2242 ms. Trace:
I20250114 02:54:30.990022 1965 rpcz_store.cc:276] 0114 02:54:28.744813 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:28.744964 (+ 151us) service_pool.cc:225] Handling call
0114 02:54:30.974387 (+2229423us) ts_tablet_manager.cc:1145] Deleting tablet 59a46bc865cc48df8e98598764fd1b54
0114 02:54:30.974421 (+ 34us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:30.987231 (+ 12810us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2228790}
W20250114 02:54:30.991374 456 catalog_manager.cc:4852] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet 59a46bc865cc48df8e98598764fd1b54 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59a46bc865cc48df8e98598764fd1b54 already in progress: creating tablet
I20250114 02:54:30.993714 1207 ts_tablet_manager.cc:1428] T c86562688d5b4fffb71cbdf7be0afc37 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.028s user 0.009s sys 0.001s
I20250114 02:54:30.998602 1992 ts_tablet_manager.cc:1918] T ed7f575653c848d7b076c17c38c4a244 P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:30.998955 1992 log.cc:1198] T ed7f575653c848d7b076c17c38c4a244 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/ed7f575653c848d7b076c17c38c4a244
I20250114 02:54:30.999539 1992 ts_tablet_manager.cc:1939] T ed7f575653c848d7b076c17c38c4a244 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:30.960304 463 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 044832851f804208b0a371eb0857dbf3 (table default.Table_0518d8bed7304c86aff32e05e6f07c0d [id=515a2fe3f1e84e82bccc02ef5cecef1b]). Expected version 1 got 0
I20250114 02:54:31.001713 463 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 92cd8eccacf34ab7a66b3bbc69df3697 (table default.Table_990b8d5f1bd146d6b358457f954780c9 [id=26627ccdc808451fb9593ad7a6df6991]). Expected version 2 got 0
I20250114 02:54:31.003751 1205 tablet_bootstrap.cc:654] T ed7f575653c848d7b076c17c38c4a244 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:31.009508 1965 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55068 (request call id 307) took 2061 ms. Trace:
I20250114 02:54:31.010175 1965 rpcz_store.cc:276] 0114 02:54:28.947887 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:28.948115 (+ 228us) service_pool.cc:225] Handling call
0114 02:54:30.992097 (+2043982us) ts_tablet_manager.cc:1145] Deleting tablet 9ffec1d1bb594fda83df8d94d4533d56
0114 02:54:30.992135 (+ 38us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:31.009453 (+ 17318us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2043344}
W20250114 02:54:31.011655 456 catalog_manager.cc:4852] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet 9ffec1d1bb594fda83df8d94d4533d56 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9ffec1d1bb594fda83df8d94d4533d56 already in progress: creating tablet
I20250114 02:54:31.014144 2151 raft_consensus.cc:491] T 141cede72b8741a8b0615d7c6bb823bb P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:31.014588 2151 raft_consensus.cc:513] T 141cede72b8741a8b0615d7c6bb823bb P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:31.016363 2151 leader_election.cc:290] T 141cede72b8741a8b0615d7c6bb823bb P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:31.018523 957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141cede72b8741a8b0615d7c6bb823bb" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
W20250114 02:54:31.019737 751 leader_election.cc:343] T 141cede72b8741a8b0615d7c6bb823bb P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:31.020620 1103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141cede72b8741a8b0615d7c6bb823bb" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
W20250114 02:54:31.021829 753 leader_election.cc:343] T 141cede72b8741a8b0615d7c6bb823bb P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:31.022154 753 leader_election.cc:304] T 141cede72b8741a8b0615d7c6bb823bb P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:31.022387 1990 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 305) took 1653 ms. Trace:
I20250114 02:54:31.022878 1207 tablet_bootstrap.cc:492] T fc406eab968745d893f1a8538ff61f37 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:31.023393 2151 raft_consensus.cc:2743] T 141cede72b8741a8b0615d7c6bb823bb P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:31.022706 1990 rpcz_store.cc:276] 0114 02:54:29.368855 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:29.374264 (+ 5409us) service_pool.cc:225] Handling call
0114 02:54:30.988447 (+1614183us) ts_tablet_manager.cc:1145] Deleting tablet 321b9ad6d1ac4d1c80c7a01faf2015af
0114 02:54:30.996281 (+ 7834us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:31.022332 (+ 26051us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1613548}
W20250114 02:54:31.025705 458 catalog_manager.cc:4839] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 321b9ad6d1ac4d1c80c7a01faf2015af because the tablet was not found. No further retry: Not found: Tablet not found: 321b9ad6d1ac4d1c80c7a01faf2015af
I20250114 02:54:31.026988 1990 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 306) took 1651 ms. Trace:
I20250114 02:54:31.027169 1965 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55068 (request call id 311) took 1684 ms. Trace:
I20250114 02:54:31.027336 1990 rpcz_store.cc:276] 0114 02:54:29.375213 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:29.375616 (+ 403us) service_pool.cc:225] Handling call
0114 02:54:31.026768 (+1651152us) ts_tablet_manager.cc:1145] Deleting tablet 266da813b4514329be0ad97d07b7114f
0114 02:54:31.026806 (+ 38us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:31.026939 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1650540}
I20250114 02:54:31.027809 1965 rpcz_store.cc:276] 0114 02:54:29.343071 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:29.343191 (+ 120us) service_pool.cc:225] Handling call
0114 02:54:31.026907 (+1683716us) ts_tablet_manager.cc:1145] Deleting tablet ca986f81ed6848a89588366f3f400eec
0114 02:54:31.026947 (+ 40us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:31.027116 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1682953}
W20250114 02:54:31.028535 458 catalog_manager.cc:4839] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 266da813b4514329be0ad97d07b7114f because the tablet was not found. No further retry: Not found: Tablet not found: 266da813b4514329be0ad97d07b7114f
W20250114 02:54:31.028966 456 catalog_manager.cc:4839] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet ca986f81ed6848a89588366f3f400eec because the tablet was not found. No further retry: Not found: Tablet not found: ca986f81ed6848a89588366f3f400eec
I20250114 02:54:31.029865 1965 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55068 (request call id 312) took 1683 ms. Trace:
I20250114 02:54:31.030144 1965 rpcz_store.cc:276] 0114 02:54:29.346175 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:29.346274 (+ 99us) service_pool.cc:225] Handling call
0114 02:54:31.029640 (+1683366us) ts_tablet_manager.cc:1145] Deleting tablet 1381e26725174b9c877075cf5444f566
0114 02:54:31.029675 (+ 35us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:31.029816 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1682753}
I20250114 02:54:31.030443 1990 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 310) took 1519 ms. Trace:
W20250114 02:54:31.031116 456 catalog_manager.cc:4839] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet 1381e26725174b9c877075cf5444f566 because the tablet was not found. No further retry: Not found: Tablet not found: 1381e26725174b9c877075cf5444f566
I20250114 02:54:31.032021 1965 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55068 (request call id 310) took 1695 ms. Trace:
I20250114 02:54:31.030853 1990 rpcz_store.cc:276] 0114 02:54:29.510853 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:29.511279 (+ 426us) service_pool.cc:225] Handling call
0114 02:54:31.030198 (+1518919us) ts_tablet_manager.cc:1145] Deleting tablet 46360f726fad4534a2daf67581346fc1
0114 02:54:31.030246 (+ 48us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:31.030395 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1517359}
I20250114 02:54:31.032322 1965 rpcz_store.cc:276] 0114 02:54:29.336592 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:29.339396 (+ 2804us) service_pool.cc:225] Handling call
0114 02:54:31.031785 (+1692389us) ts_tablet_manager.cc:1145] Deleting tablet 5ac7708550954a209e2a8d217c3b39bd
0114 02:54:31.031821 (+ 36us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:31.031977 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1682395}
W20250114 02:54:31.032871 458 catalog_manager.cc:4839] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 46360f726fad4534a2daf67581346fc1 because the tablet was not found. No further retry: Not found: Tablet not found: 46360f726fad4534a2daf67581346fc1
W20250114 02:54:31.033254 456 catalog_manager.cc:4839] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet 5ac7708550954a209e2a8d217c3b39bd because the tablet was not found. No further retry: Not found: Tablet not found: 5ac7708550954a209e2a8d217c3b39bd
I20250114 02:54:31.033740 1990 tablet_replica.cc:331] T e15e6e52d7294aeeb4463b3a76022ec3 P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:31.035012 469 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet 41814f82964a4bcb8dbb6d3738052448
I20250114 02:54:31.036752 931 tablet_service.cc:1514] Processing DeleteTablet for tablet 41814f82964a4bcb8dbb6d3738052448 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:30 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:31.036897 469 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_161092dfc9464fae90944e56b7ba1740"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:31.038277 469 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_161092dfc9464fae90944e56b7ba1740 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:31.036487 1084 tablet_service.cc:1514] Processing DeleteTablet for tablet 41814f82964a4bcb8dbb6d3738052448 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:30 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:31.038671 794 tablet_service.cc:1514] Processing DeleteTablet for tablet 41814f82964a4bcb8dbb6d3738052448 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:30 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:31.039613 465 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet 587a5f39282a40d0a8c1591fdf36f341
I20250114 02:54:31.040191 1990 raft_consensus.cc:2238] T e15e6e52d7294aeeb4463b3a76022ec3 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:31.041345 1084 tablet_service.cc:1514] Processing DeleteTablet for tablet 587a5f39282a40d0a8c1591fdf36f341 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:30 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:31.041460 465 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_d0eabdfdf4214a32b3afc2c70c63eb9c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:31.042820 465 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d0eabdfdf4214a32b3afc2c70c63eb9c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:31.048732 931 tablet_service.cc:1514] Processing DeleteTablet for tablet 587a5f39282a40d0a8c1591fdf36f341 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:30 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:31.049265 1207 tablet_bootstrap.cc:654] T fc406eab968745d893f1a8538ff61f37 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:31.056002 2151 raft_consensus.cc:491] T df045d80d8ce4ff194a6d2039a44bce6 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:31.056644 2151 raft_consensus.cc:513] T df045d80d8ce4ff194a6d2039a44bce6 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:31.059221 1990 raft_consensus.cc:2267] T e15e6e52d7294aeeb4463b3a76022ec3 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:31.059096 794 tablet_service.cc:1514] Processing DeleteTablet for tablet 587a5f39282a40d0a8c1591fdf36f341 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:30 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:31.061048 1103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df045d80d8ce4ff194a6d2039a44bce6" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:31.060045 2143 raft_consensus.cc:491] T 6e858a8449444ef999b9bbef612249d3 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:31.061830 1992 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 244) took 3842 ms. Trace:
I20250114 02:54:31.061828 2143 raft_consensus.cc:513] T 6e858a8449444ef999b9bbef612249d3 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:31.063863 2143 leader_election.cc:290] T 6e858a8449444ef999b9bbef612249d3 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:31.064405 2151 leader_election.cc:290] T df045d80d8ce4ff194a6d2039a44bce6 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:31.065786 1103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e858a8449444ef999b9bbef612249d3" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:31.066931 1990 ts_tablet_manager.cc:1905] T e15e6e52d7294aeeb4463b3a76022ec3 P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:31.067523 815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e858a8449444ef999b9bbef612249d3" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
W20250114 02:54:31.069420 753 leader_election.cc:343] T df045d80d8ce4ff194a6d2039a44bce6 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:31.070447 898 leader_election.cc:304] T 6e858a8449444ef999b9bbef612249d3 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 525198593c8e47afaee54e74267b87e3; no voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:31.071712 2143 raft_consensus.cc:2743] T 6e858a8449444ef999b9bbef612249d3 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:31.062156 1992 rpcz_store.cc:276] 0114 02:54:27.219149 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:27.219271 (+ 122us) service_pool.cc:225] Handling call
0114 02:54:30.747230 (+3527959us) ts_tablet_manager.cc:1145] Deleting tablet ed7f575653c848d7b076c17c38c4a244
0114 02:54:30.747655 (+ 425us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:30.847760 (+100105us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:30.998384 (+150624us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:31.061760 (+ 63376us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11303,"tablet-delete.queue_time_us":3527445}
I20250114 02:54:31.073158 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet ed7f575653c848d7b076c17c38c4a244 (table default.Table_f842a61f536940588cd46bedc206f2b3 [id=954669573a3c4a97a488805e05b0b220]) successfully deleted
I20250114 02:54:31.075342 1992 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 248) took 3474 ms. Trace:
I20250114 02:54:31.075675 1992 rpcz_store.cc:276] 0114 02:54:27.600681 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:27.600829 (+ 148us) service_pool.cc:225] Handling call
0114 02:54:31.072804 (+3471975us) ts_tablet_manager.cc:1145] Deleting tablet 0b8361cd538046e0bcb4aac821748f67
0114 02:54:31.072845 (+ 41us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:31.075269 (+ 2424us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3471356}
I20250114 02:54:31.044502 1205 tablet_bootstrap.cc:492] T ed7f575653c848d7b076c17c38c4a244 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:31.076754 1992 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 249) took 3468 ms. Trace:
I20250114 02:54:31.076843 1205 ts_tablet_manager.cc:1397] T ed7f575653c848d7b076c17c38c4a244 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.119s user 0.005s sys 0.008s
I20250114 02:54:31.077016 1992 rpcz_store.cc:276] 0114 02:54:27.608307 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:27.625499 (+ 17192us) service_pool.cc:225] Handling call
0114 02:54:31.076439 (+3450940us) ts_tablet_manager.cc:1145] Deleting tablet ddad7584337345109ce102702008ab37
0114 02:54:31.076477 (+ 38us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:31.076711 (+ 234us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3328879}
W20250114 02:54:31.079162 458 catalog_manager.cc:4852] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet 0b8361cd538046e0bcb4aac821748f67 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0b8361cd538046e0bcb4aac821748f67 already in progress: creating tablet
W20250114 02:54:31.079888 458 catalog_manager.cc:4852] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet ddad7584337345109ce102702008ab37 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ddad7584337345109ce102702008ab37 already in progress: creating tablet
I20250114 02:54:31.089056 1205 raft_consensus.cc:357] T ed7f575653c848d7b076c17c38c4a244 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:31.089732 1205 raft_consensus.cc:383] T ed7f575653c848d7b076c17c38c4a244 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:31.089965 1205 raft_consensus.cc:738] T ed7f575653c848d7b076c17c38c4a244 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:31.081800 1206 tablet_bootstrap.cc:654] T 30a56fb2d941470590e17f37b90aef26 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:31.095124 1992 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 256) took 3037 ms. Trace:
I20250114 02:54:31.090505 1205 consensus_queue.cc:260] T ed7f575653c848d7b076c17c38c4a244 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:31.105114 1992 rpcz_store.cc:276] 0114 02:54:28.057713 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:28.057841 (+ 128us) service_pool.cc:225] Handling call
0114 02:54:31.077872 (+3020031us) ts_tablet_manager.cc:1145] Deleting tablet d56255ca806d4e2cac09fbb5e1b1efbd
0114 02:54:31.077907 (+ 35us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:31.095061 (+ 17154us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2745486}
W20250114 02:54:31.106446 458 catalog_manager.cc:4852] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet d56255ca806d4e2cac09fbb5e1b1efbd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d56255ca806d4e2cac09fbb5e1b1efbd already in progress: creating tablet
I20250114 02:54:31.107606 1992 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 257) took 3049 ms. Trace:
I20250114 02:54:31.107959 1992 rpcz_store.cc:276] 0114 02:54:28.057999 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:28.058166 (+ 167us) service_pool.cc:225] Handling call
0114 02:54:31.107280 (+3049114us) ts_tablet_manager.cc:1145] Deleting tablet 992bb16197b74be5b1dc884f29d9c240
0114 02:54:31.107321 (+ 41us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:31.107554 (+ 233us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2774723}
W20250114 02:54:31.109136 458 catalog_manager.cc:4852] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet 992bb16197b74be5b1dc884f29d9c240 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 992bb16197b74be5b1dc884f29d9c240 already in progress: creating tablet
I20250114 02:54:31.109997 1992 tablet_replica.cc:331] T 59a46bc865cc48df8e98598764fd1b54 P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:31.111858 1992 raft_consensus.cc:2238] T 59a46bc865cc48df8e98598764fd1b54 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:31.114317 1992 raft_consensus.cc:2267] T 59a46bc865cc48df8e98598764fd1b54 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:31.120646 1205 ts_tablet_manager.cc:1428] T ed7f575653c848d7b076c17c38c4a244 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.044s user 0.006s sys 0.001s
I20250114 02:54:31.121793 1992 ts_tablet_manager.cc:1905] T 59a46bc865cc48df8e98598764fd1b54 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:31.122772 1206 tablet_bootstrap.cc:492] T 30a56fb2d941470590e17f37b90aef26 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:31.124286 2143 raft_consensus.cc:491] T 6e2fb05f72b743e2a02ed36b63f80b23 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:31.124712 2143 raft_consensus.cc:513] T 6e2fb05f72b743e2a02ed36b63f80b23 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:31.126451 2143 leader_election.cc:290] T 6e2fb05f72b743e2a02ed36b63f80b23 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:31.128473 811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e2fb05f72b743e2a02ed36b63f80b23" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:31.129104 811 raft_consensus.cc:2463] T 6e2fb05f72b743e2a02ed36b63f80b23 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:31.129091 1103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e2fb05f72b743e2a02ed36b63f80b23" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:31.130249 898 leader_election.cc:304] T 6e2fb05f72b743e2a02ed36b63f80b23 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
W20250114 02:54:31.130996 898 leader_election.cc:343] T 6e2fb05f72b743e2a02ed36b63f80b23 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:31.131151 2143 raft_consensus.cc:2798] T 6e2fb05f72b743e2a02ed36b63f80b23 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:31.131453 2143 raft_consensus.cc:491] T 6e2fb05f72b743e2a02ed36b63f80b23 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:31.131731 2143 raft_consensus.cc:3054] T 6e2fb05f72b743e2a02ed36b63f80b23 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:31.144176 957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df045d80d8ce4ff194a6d2039a44bce6" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:31.144434 960 raft_consensus.cc:3054] T 6e858a8449444ef999b9bbef612249d3 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:31.144793 957 raft_consensus.cc:2463] T df045d80d8ce4ff194a6d2039a44bce6 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 0.
I20250114 02:54:31.146275 1965 tablet_replica.cc:331] T 84738a6c78b24c1fb64013048763b18f P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:31.147027 1965 raft_consensus.cc:2238] T 84738a6c78b24c1fb64013048763b18f P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:31.147487 1965 raft_consensus.cc:2267] T 84738a6c78b24c1fb64013048763b18f P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:31.148667 751 leader_election.cc:304] T df045d80d8ce4ff194a6d2039a44bce6 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters: 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:31.149633 2151 raft_consensus.cc:2798] T df045d80d8ce4ff194a6d2039a44bce6 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:31.149924 2151 raft_consensus.cc:491] T df045d80d8ce4ff194a6d2039a44bce6 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:31.150178 2151 raft_consensus.cc:3054] T df045d80d8ce4ff194a6d2039a44bce6 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:31.166921 469 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_d21051d46c4043488adb2d04cfb8cead"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:31.168471 469 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d21051d46c4043488adb2d04cfb8cead in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:31.172269 465 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_1d46f092cec5487fbf47ab3d83aaeb87"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:31.173604 465 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1d46f092cec5487fbf47ab3d83aaeb87 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250114 02:54:31.177561 1040 consensus_peers.cc:487] T 84738a6c78b24c1fb64013048763b18f P 90d6c2c58bb54e0eafb365625ac33871 -> Peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Couldn't send request to peer 525198593c8e47afaee54e74267b87e3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250114 02:54:31.166816 2151 raft_consensus.cc:513] T df045d80d8ce4ff194a6d2039a44bce6 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:31.184006 2151 leader_election.cc:290] T df045d80d8ce4ff194a6d2039a44bce6 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Requested vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:31.184276 959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df045d80d8ce4ff194a6d2039a44bce6" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
I20250114 02:54:31.184751 959 raft_consensus.cc:3054] T df045d80d8ce4ff194a6d2039a44bce6 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:31.142614 2143 raft_consensus.cc:513] T 6e2fb05f72b743e2a02ed36b63f80b23 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:31.186090 2143 leader_election.cc:290] T 6e2fb05f72b743e2a02ed36b63f80b23 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Requested vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:31.187024 1206 ts_tablet_manager.cc:1397] T 30a56fb2d941470590e17f37b90aef26 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.205s user 0.010s sys 0.000s
I20250114 02:54:31.190011 1207 tablet_bootstrap.cc:492] T fc406eab968745d893f1a8538ff61f37 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:31.190163 1102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e2fb05f72b743e2a02ed36b63f80b23" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:31.190464 1207 ts_tablet_manager.cc:1397] T fc406eab968745d893f1a8538ff61f37 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.168s user 0.012s sys 0.011s
I20250114 02:54:31.190523 1103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df045d80d8ce4ff194a6d2039a44bce6" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
W20250114 02:54:31.191483 898 leader_election.cc:343] T 6e2fb05f72b743e2a02ed36b63f80b23 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:31.192639 1205 tablet_bootstrap.cc:492] T 3161f75bc4e84000af022faa9804c810 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:31.192687 1207 raft_consensus.cc:357] T fc406eab968745d893f1a8538ff61f37 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:31.193284 1207 raft_consensus.cc:383] T fc406eab968745d893f1a8538ff61f37 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:31.193513 1207 raft_consensus.cc:738] T fc406eab968745d893f1a8538ff61f37 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:31.195253 959 raft_consensus.cc:2463] T df045d80d8ce4ff194a6d2039a44bce6 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 1.
I20250114 02:54:31.196063 751 leader_election.cc:304] T df045d80d8ce4ff194a6d2039a44bce6 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
I20250114 02:54:31.196916 2151 raft_consensus.cc:2798] T df045d80d8ce4ff194a6d2039a44bce6 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election won for term 1
W20250114 02:54:31.197255 753 leader_election.cc:343] T df045d80d8ce4ff194a6d2039a44bce6 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:31.198446 811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e2fb05f72b743e2a02ed36b63f80b23" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:31.198925 811 raft_consensus.cc:3054] T 6e2fb05f72b743e2a02ed36b63f80b23 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:31.204730 1965 ts_tablet_manager.cc:1905] T 84738a6c78b24c1fb64013048763b18f P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:31.208014 1206 raft_consensus.cc:357] T 30a56fb2d941470590e17f37b90aef26 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:31.208732 1206 raft_consensus.cc:383] T 30a56fb2d941470590e17f37b90aef26 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:31.208614 465 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_666fde8ba59a4b688d83a1a27adfb7f7" } new_table_name: "default.Table_bbdaf710a91a429f9dd60f183ceea9f9" modify_external_catalogs: true
I20250114 02:54:31.208977 1206 raft_consensus.cc:738] T 30a56fb2d941470590e17f37b90aef26 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:31.211077 811 raft_consensus.cc:2463] T 6e2fb05f72b743e2a02ed36b63f80b23 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:31.210640 1206 consensus_queue.cc:260] T 30a56fb2d941470590e17f37b90aef26 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:31.212347 898 leader_election.cc:304] T 6e2fb05f72b743e2a02ed36b63f80b23 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters: 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:31.213021 2097 raft_consensus.cc:2798] T 6e2fb05f72b743e2a02ed36b63f80b23 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:31.212879 1207 consensus_queue.cc:260] T fc406eab968745d893f1a8538ff61f37 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:31.231987 2151 raft_consensus.cc:695] T df045d80d8ce4ff194a6d2039a44bce6 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 LEADER]: Becoming Leader. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Running, Role: LEADER
I20250114 02:54:31.236781 2151 consensus_queue.cc:237] T df045d80d8ce4ff194a6d2039a44bce6 P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:31.238758 1103 raft_consensus.cc:3054] T a52d6749ff834b62a8e20a2f45935240 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:31.242254 2097 raft_consensus.cc:695] T 6e2fb05f72b743e2a02ed36b63f80b23 P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Becoming Leader. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Running, Role: LEADER
I20250114 02:54:31.243448 469 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_b146341dc9614d3d92e92e0ecfe60405" } modify_external_catalogs: true
I20250114 02:54:31.243959 469 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_b146341dc9614d3d92e92e0ecfe60405" } modify_external_catalogs: true
I20250114 02:54:31.244017 2097 consensus_queue.cc:237] T 6e2fb05f72b743e2a02ed36b63f80b23 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:31.247877 2143 consensus_queue.cc:1035] T a52d6749ff834b62a8e20a2f45935240 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:31.249449 1207 ts_tablet_manager.cc:1428] T fc406eab968745d893f1a8538ff61f37 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.059s user 0.005s sys 0.000s
I20250114 02:54:31.250376 1207 tablet_bootstrap.cc:492] T df045d80d8ce4ff194a6d2039a44bce6 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:31.251400 2201 raft_consensus.cc:491] T fc406eab968745d893f1a8538ff61f37 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:31.251818 2201 raft_consensus.cc:513] T fc406eab968745d893f1a8538ff61f37 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:31.253533 2201 leader_election.cc:290] T fc406eab968745d893f1a8538ff61f37 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:31.254786 813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc406eab968745d893f1a8538ff61f37" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:31.255106 959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc406eab968745d893f1a8538ff61f37" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:31.255373 813 raft_consensus.cc:2463] T fc406eab968745d893f1a8538ff61f37 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 0.
I20250114 02:54:31.256047 1206 ts_tablet_manager.cc:1428] T 30a56fb2d941470590e17f37b90aef26 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.069s user 0.005s sys 0.000s
W20250114 02:54:31.256141 1040 leader_election.cc:343] T fc406eab968745d893f1a8538ff61f37 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:31.256657 1042 leader_election.cc:304] T fc406eab968745d893f1a8538ff61f37 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters: 525198593c8e47afaee54e74267b87e3
I20250114 02:54:31.257529 2201 raft_consensus.cc:2798] T fc406eab968745d893f1a8538ff61f37 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:31.257840 2201 raft_consensus.cc:491] T fc406eab968745d893f1a8538ff61f37 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:31.258194 2201 raft_consensus.cc:3054] T fc406eab968745d893f1a8538ff61f37 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:31.264355 1205 tablet_bootstrap.cc:654] T 3161f75bc4e84000af022faa9804c810 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:31.265326 2201 raft_consensus.cc:513] T fc406eab968745d893f1a8538ff61f37 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:31.268428 959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc406eab968745d893f1a8538ff61f37" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
W20250114 02:54:31.269598 1040 leader_election.cc:343] T fc406eab968745d893f1a8538ff61f37 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
W20250114 02:54:31.284653 465 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6027bc3f9121401fbdced24aabaa2d94 has no consensus state
I20250114 02:54:31.290155 1205 tablet_bootstrap.cc:492] T 3161f75bc4e84000af022faa9804c810 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:31.290957 1205 ts_tablet_manager.cc:1397] T 3161f75bc4e84000af022faa9804c810 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.098s user 0.009s sys 0.001s
I20250114 02:54:31.291844 465 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_ba8866c1c90041268e09ff59a46503f1" } modify_external_catalogs: true
I20250114 02:54:31.292349 465 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_ba8866c1c90041268e09ff59a46503f1" } modify_external_catalogs: true
I20250114 02:54:31.298568 1992 ts_tablet_manager.cc:1918] T 59a46bc865cc48df8e98598764fd1b54 P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:31.299000 1992 log.cc:1198] T 59a46bc865cc48df8e98598764fd1b54 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/59a46bc865cc48df8e98598764fd1b54
I20250114 02:54:31.299640 1992 ts_tablet_manager.cc:1939] T 59a46bc865cc48df8e98598764fd1b54 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:31.302148 1205 raft_consensus.cc:357] T 3161f75bc4e84000af022faa9804c810 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:31.302816 1205 raft_consensus.cc:383] T 3161f75bc4e84000af022faa9804c810 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:31.303035 1205 raft_consensus.cc:738] T 3161f75bc4e84000af022faa9804c810 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:31.303746 1990 ts_tablet_manager.cc:1918] T e15e6e52d7294aeeb4463b3a76022ec3 P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:31.303546 1205 consensus_queue.cc:260] T 3161f75bc4e84000af022faa9804c810 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:31.306519 1206 tablet_bootstrap.cc:492] T 967f4d5b944543d49e87645d069b5b55 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:31.317054 1205 ts_tablet_manager.cc:1428] T 3161f75bc4e84000af022faa9804c810 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.026s user 0.007s sys 0.000s
I20250114 02:54:31.317698 1992 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 265) took 2570 ms. Trace:
I20250114 02:54:31.318039 1992 rpcz_store.cc:276] 0114 02:54:28.747601 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:28.747784 (+ 183us) service_pool.cc:225] Handling call
0114 02:54:31.109870 (+2362086us) ts_tablet_manager.cc:1145] Deleting tablet 59a46bc865cc48df8e98598764fd1b54
0114 02:54:31.109905 (+ 35us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:31.141620 (+ 31715us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:31.298357 (+156737us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:31.317636 (+ 19279us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2361330}
I20250114 02:54:31.319113 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet 59a46bc865cc48df8e98598764fd1b54 (table default.Table_1097f09fb10e441fb639f8d7aa12d487 [id=d8c9691818d042d7b49475fa009c4ca1]) successfully deleted
W20250114 02:54:31.320078 458 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 6027bc3f9121401fbdced24aabaa2d94 (table default.Table_bbdaf710a91a429f9dd60f183ceea9f9 [id=4507c028bd074fd0a1d0231ea671b355], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 6027bc3f9121401fbdced24aabaa2d94 has no consensus state
I20250114 02:54:31.323585 2143 raft_consensus.cc:491] T 3161f75bc4e84000af022faa9804c810 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:31.324030 2143 raft_consensus.cc:513] T 3161f75bc4e84000af022faa9804c810 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:31.325870 2143 leader_election.cc:290] T 3161f75bc4e84000af022faa9804c810 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:31.328569 811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3161f75bc4e84000af022faa9804c810" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:31.329252 811 raft_consensus.cc:2463] T 3161f75bc4e84000af022faa9804c810 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:31.332851 1992 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 274) took 2393 ms. Trace:
I20250114 02:54:31.333159 1992 rpcz_store.cc:276] 0114 02:54:28.939697 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:28.940172 (+ 475us) service_pool.cc:225] Handling call
0114 02:54:31.332524 (+2392352us) ts_tablet_manager.cc:1145] Deleting tablet 9ffec1d1bb594fda83df8d94d4533d56
0114 02:54:31.332572 (+ 48us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:31.332801 (+ 229us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2391813}
W20250114 02:54:31.334200 458 catalog_manager.cc:4852] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet 9ffec1d1bb594fda83df8d94d4533d56 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9ffec1d1bb594fda83df8d94d4533d56 already in progress: creating tablet
I20250114 02:54:31.339239 469 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet a1cf92e5275b454ca893b12f2d90d25b
I20250114 02:54:31.339012 1992 tablet_replica.cc:331] T e15e6e52d7294aeeb4463b3a76022ec3 P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:31.342435 794 tablet_service.cc:1514] Processing DeleteTablet for tablet a1cf92e5275b454ca893b12f2d90d25b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:31 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:31.349225 1205 tablet_bootstrap.cc:492] T 59a46bc865cc48df8e98598764fd1b54 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:31.354928 1205 tablet_bootstrap.cc:654] T 59a46bc865cc48df8e98598764fd1b54 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:31.355013 898 leader_election.cc:304] T 3161f75bc4e84000af022faa9804c810 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
I20250114 02:54:31.356889 1992 raft_consensus.cc:2238] T e15e6e52d7294aeeb4463b3a76022ec3 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:31.359763 2248 raft_consensus.cc:2798] T 3161f75bc4e84000af022faa9804c810 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:31.360378 2248 raft_consensus.cc:491] T 3161f75bc4e84000af022faa9804c810 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:31.360967 2248 raft_consensus.cc:3054] T 3161f75bc4e84000af022faa9804c810 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:31.360136 1992 raft_consensus.cc:2267] T e15e6e52d7294aeeb4463b3a76022ec3 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:31.382836 1992 ts_tablet_manager.cc:1905] T e15e6e52d7294aeeb4463b3a76022ec3 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:31.387766 811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc406eab968745d893f1a8538ff61f37" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:31.388532 811 raft_consensus.cc:3054] T fc406eab968745d893f1a8538ff61f37 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:31.304087 1990 log.cc:1198] T e15e6e52d7294aeeb4463b3a76022ec3 P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/e15e6e52d7294aeeb4463b3a76022ec3
I20250114 02:54:31.402235 465 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet 2837990edc4d4bea871a8114533c9352
I20250114 02:54:31.404986 1206 tablet_bootstrap.cc:654] T 967f4d5b944543d49e87645d069b5b55 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:31.405462 794 tablet_service.cc:1514] Processing DeleteTablet for tablet 2837990edc4d4bea871a8114533c9352 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:31 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:31.345412 1103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3161f75bc4e84000af022faa9804c810" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:31.386368 2201 leader_election.cc:290] T fc406eab968745d893f1a8538ff61f37 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Requested vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:31.404533 1084 tablet_service.cc:1514] Processing DeleteTablet for tablet 2837990edc4d4bea871a8114533c9352 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:31 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:31.399137 1990 ts_tablet_manager.cc:1939] T e15e6e52d7294aeeb4463b3a76022ec3 P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
I20250114 02:54:31.322222 1207 tablet_bootstrap.cc:654] T df045d80d8ce4ff194a6d2039a44bce6 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
W20250114 02:54:31.422001 898 leader_election.cc:343] T 3161f75bc4e84000af022faa9804c810 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:31.429075 1206 tablet_bootstrap.cc:492] T 967f4d5b944543d49e87645d069b5b55 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:31.421155 2143 consensus_queue.cc:1035] T a52d6749ff834b62a8e20a2f45935240 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20250114 02:54:31.430709 1206 ts_tablet_manager.cc:1397] T 967f4d5b944543d49e87645d069b5b55 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.157s user 0.021s sys 0.007s
I20250114 02:54:31.432396 811 raft_consensus.cc:2463] T fc406eab968745d893f1a8538ff61f37 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1.
I20250114 02:54:31.438341 1205 tablet_bootstrap.cc:492] T 59a46bc865cc48df8e98598764fd1b54 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:31.439213 1205 ts_tablet_manager.cc:1397] T 59a46bc865cc48df8e98598764fd1b54 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.090s user 0.007s sys 0.004s
I20250114 02:54:31.447220 2248 raft_consensus.cc:513] T 3161f75bc4e84000af022faa9804c810 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:31.447625 1042 leader_election.cc:304] T fc406eab968745d893f1a8538ff61f37 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters: 525198593c8e47afaee54e74267b87e3
I20250114 02:54:31.448385 2201 raft_consensus.cc:2798] T fc406eab968745d893f1a8538ff61f37 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:31.449249 2201 raft_consensus.cc:695] T fc406eab968745d893f1a8538ff61f37 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Becoming Leader. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Running, Role: LEADER
I20250114 02:54:31.450066 2201 consensus_queue.cc:237] T fc406eab968745d893f1a8538ff61f37 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:31.453212 1990 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 309) took 1942 ms. Trace:
I20250114 02:54:31.439960 1206 raft_consensus.cc:357] T 967f4d5b944543d49e87645d069b5b55 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:31.457052 1206 raft_consensus.cc:383] T 967f4d5b944543d49e87645d069b5b55 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:31.457790 1990 rpcz_store.cc:276] 0114 02:54:29.510507 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:29.516354 (+ 5847us) service_pool.cc:225] Handling call
0114 02:54:31.033608 (+1517254us) ts_tablet_manager.cc:1145] Deleting tablet e15e6e52d7294aeeb4463b3a76022ec3
0114 02:54:31.033641 (+ 33us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:31.235111 (+201470us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:31.303555 (+ 68444us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:31.453159 (+149604us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1516647}
I20250114 02:54:31.460016 1206 raft_consensus.cc:738] T 967f4d5b944543d49e87645d069b5b55 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:31.462457 1206 consensus_queue.cc:260] T 967f4d5b944543d49e87645d069b5b55 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:31.461421 1084 tablet_service.cc:1514] Processing DeleteTablet for tablet a1cf92e5275b454ca893b12f2d90d25b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:31 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:31.463078 469 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_36be92916bc1421bb2c3261d44b3a6db"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:31.464166 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet e15e6e52d7294aeeb4463b3a76022ec3 (table default.Table_d1de82caa5b94791b6c7dfed1c376e8a [id=03c775a54e9e443cbc4fab853bd6d49c]) successfully deleted
W20250114 02:54:31.464713 469 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_36be92916bc1421bb2c3261d44b3a6db in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:31.465698 1990 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 312) took 1924 ms. Trace:
I20250114 02:54:31.473016 1990 rpcz_store.cc:276] 0114 02:54:29.541731 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:29.544326 (+ 2595us) service_pool.cc:225] Handling call
0114 02:54:31.465419 (+1921093us) ts_tablet_manager.cc:1145] Deleting tablet f0bd9bd5a17d418d8082ed0ebd1bf4cc
0114 02:54:31.465466 (+ 47us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:31.465645 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1920512}
I20250114 02:54:31.475203 1206 ts_tablet_manager.cc:1428] T 967f4d5b944543d49e87645d069b5b55 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.044s user 0.005s sys 0.000s
W20250114 02:54:31.477223 458 catalog_manager.cc:4839] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet f0bd9bd5a17d418d8082ed0ebd1bf4cc because the tablet was not found. No further retry: Not found: Tablet not found: f0bd9bd5a17d418d8082ed0ebd1bf4cc
I20250114 02:54:31.478395 1990 tablet_replica.cc:331] T 25e85b0c5c9a4a3ca35a001359f21167 P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:31.480239 1990 raft_consensus.cc:2238] T 25e85b0c5c9a4a3ca35a001359f21167 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:31.455461 931 tablet_service.cc:1514] Processing DeleteTablet for tablet 2837990edc4d4bea871a8114533c9352 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:31 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:31.456152 1009 heartbeater.cc:346] Connected to a master server at 127.0.107.126:45367
I20250114 02:54:31.482622 2248 leader_election.cc:290] T 3161f75bc4e84000af022faa9804c810 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Requested vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:31.485581 1990 raft_consensus.cc:2267] T 25e85b0c5c9a4a3ca35a001359f21167 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:31.487360 1990 ts_tablet_manager.cc:1905] T 25e85b0c5c9a4a3ca35a001359f21167 P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:31.460569 932 tablet_service.cc:1514] Processing DeleteTablet for tablet a1cf92e5275b454ca893b12f2d90d25b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:31 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:31.463230 1965 ts_tablet_manager.cc:1918] T 84738a6c78b24c1fb64013048763b18f P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:31.477370 1205 raft_consensus.cc:357] T 59a46bc865cc48df8e98598764fd1b54 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:31.488790 1965 log.cc:1198] T 84738a6c78b24c1fb64013048763b18f P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/84738a6c78b24c1fb64013048763b18f
I20250114 02:54:31.488878 1205 raft_consensus.cc:383] T 59a46bc865cc48df8e98598764fd1b54 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:31.489157 1205 raft_consensus.cc:738] T 59a46bc865cc48df8e98598764fd1b54 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:31.489542 1965 ts_tablet_manager.cc:1939] T 84738a6c78b24c1fb64013048763b18f P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:31.490249 1965 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55068 (request call id 313) took 2142 ms. Trace:
I20250114 02:54:31.490422 811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3161f75bc4e84000af022faa9804c810" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:31.490551 1965 rpcz_store.cc:276] 0114 02:54:29.347756 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:29.348917 (+ 1161us) service_pool.cc:225] Handling call
0114 02:54:31.034465 (+1685548us) ts_tablet_manager.cc:1145] Deleting tablet 84738a6c78b24c1fb64013048763b18f
0114 02:54:31.143914 (+109449us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:31.247225 (+103311us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:31.428010 (+180785us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:31.490190 (+ 62180us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1684659}
I20250114 02:54:31.491007 811 raft_consensus.cc:3054] T 3161f75bc4e84000af022faa9804c810 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:31.490988 1103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3161f75bc4e84000af022faa9804c810" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:31.491639 1965 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55068 (request call id 315) took 2104 ms. Trace:
I20250114 02:54:31.492355 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet 84738a6c78b24c1fb64013048763b18f (table default.Table_7058838b6bcb427786778096cf6f52b3 [id=b5d3461f83da46faaa772376a318b6b6]) successfully deleted
W20250114 02:54:31.492748 898 leader_election.cc:343] T 3161f75bc4e84000af022faa9804c810 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:31.492898 1206 tablet_bootstrap.cc:492] T 0b8989d340fa4ae992ef97e226d41110 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:31.489734 1205 consensus_queue.cc:260] T 59a46bc865cc48df8e98598764fd1b54 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:31.505126 1205 ts_tablet_manager.cc:1428] T 59a46bc865cc48df8e98598764fd1b54 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.065s user 0.006s sys 0.000s
I20250114 02:54:31.506533 1205 tablet_bootstrap.cc:492] T fc406eab968745d893f1a8538ff61f37 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:31.512163 1207 tablet_bootstrap.cc:492] T df045d80d8ce4ff194a6d2039a44bce6 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:31.512544 1207 ts_tablet_manager.cc:1397] T df045d80d8ce4ff194a6d2039a44bce6 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.262s user 0.009s sys 0.001s
I20250114 02:54:31.512924 811 raft_consensus.cc:2463] T 3161f75bc4e84000af022faa9804c810 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:31.514443 898 leader_election.cc:304] T 3161f75bc4e84000af022faa9804c810 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters: 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:31.515141 2097 raft_consensus.cc:2798] T 3161f75bc4e84000af022faa9804c810 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:31.515553 2097 raft_consensus.cc:695] T 3161f75bc4e84000af022faa9804c810 P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Becoming Leader. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Running, Role: LEADER
I20250114 02:54:31.516430 2097 consensus_queue.cc:237] T 3161f75bc4e84000af022faa9804c810 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:31.518065 1206 tablet_bootstrap.cc:654] T 0b8989d340fa4ae992ef97e226d41110 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:31.541110 1205 tablet_bootstrap.cc:654] T fc406eab968745d893f1a8538ff61f37 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:31.491904 1965 rpcz_store.cc:276] 0114 02:54:29.386990 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:29.387093 (+ 103us) service_pool.cc:225] Handling call
0114 02:54:31.491416 (+2104323us) ts_tablet_manager.cc:1145] Deleting tablet d3fb28717c6f4844af84bb817fe7dfc4
0114 02:54:31.491453 (+ 37us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:31.491592 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2103640}
I20250114 02:54:31.550173 1965 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55068 (request call id 314) took 2187 ms. Trace:
W20250114 02:54:31.550328 456 catalog_manager.cc:4839] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet d3fb28717c6f4844af84bb817fe7dfc4 because the tablet was not found. No further retry: Not found: Tablet not found: d3fb28717c6f4844af84bb817fe7dfc4
I20250114 02:54:31.514626 1207 raft_consensus.cc:357] T df045d80d8ce4ff194a6d2039a44bce6 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:31.550621 1965 rpcz_store.cc:276] 0114 02:54:29.362454 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:29.364313 (+ 1859us) service_pool.cc:225] Handling call
0114 02:54:31.549915 (+2185602us) ts_tablet_manager.cc:1145] Deleting tablet 46360f726fad4534a2daf67581346fc1
0114 02:54:31.549958 (+ 43us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:31.550113 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2142506}
I20250114 02:54:31.551651 1965 tablet_replica.cc:331] T e15e6e52d7294aeeb4463b3a76022ec3 P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
W20250114 02:54:31.551852 456 catalog_manager.cc:4839] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet 46360f726fad4534a2daf67581346fc1 because the tablet was not found. No further retry: Not found: Tablet not found: 46360f726fad4534a2daf67581346fc1
I20250114 02:54:31.551604 1207 raft_consensus.cc:383] T df045d80d8ce4ff194a6d2039a44bce6 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:31.556388 1207 raft_consensus.cc:738] T df045d80d8ce4ff194a6d2039a44bce6 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:31.562925 1206 tablet_bootstrap.cc:492] T 0b8989d340fa4ae992ef97e226d41110 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:31.563452 1206 ts_tablet_manager.cc:1397] T 0b8989d340fa4ae992ef97e226d41110 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.080s user 0.008s sys 0.004s
I20250114 02:54:31.564347 1207 consensus_queue.cc:260] T df045d80d8ce4ff194a6d2039a44bce6 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
W20250114 02:54:31.572360 898 consensus_peers.cc:487] T e15e6e52d7294aeeb4463b3a76022ec3 P 525198593c8e47afaee54e74267b87e3 -> Peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Couldn't send request to peer 90d6c2c58bb54e0eafb365625ac33871. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e15e6e52d7294aeeb4463b3a76022ec3. This is attempt 1: this message will repeat every 5th retry.
I20250114 02:54:31.578956 1990 ts_tablet_manager.cc:1918] T 25e85b0c5c9a4a3ca35a001359f21167 P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:31.580471 1206 raft_consensus.cc:357] T 0b8989d340fa4ae992ef97e226d41110 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:31.587848 1206 raft_consensus.cc:383] T 0b8989d340fa4ae992ef97e226d41110 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:31.588587 1206 raft_consensus.cc:738] T 0b8989d340fa4ae992ef97e226d41110 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:31.589767 1206 consensus_queue.cc:260] T 0b8989d340fa4ae992ef97e226d41110 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:31.596657 932 tablet_service.cc:1467] Processing CreateTablet for tablet 4a1d8332ecef44829642358719004d99 (DEFAULT_TABLE table=default.Table_2c307103a2e8474b9c7e18b5691f61d4 [id=c65db2976e214ac2b96b15b7a49487c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:31.597712 932 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a1d8332ecef44829642358719004d99. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:31.597911 794 tablet_service.cc:1467] Processing CreateTablet for tablet 4a1d8332ecef44829642358719004d99 (DEFAULT_TABLE table=default.Table_2c307103a2e8474b9c7e18b5691f61d4 [id=c65db2976e214ac2b96b15b7a49487c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:31.599002 794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a1d8332ecef44829642358719004d99. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:31.601683 1206 ts_tablet_manager.cc:1428] T 0b8989d340fa4ae992ef97e226d41110 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.038s user 0.005s sys 0.000s
I20250114 02:54:31.595752 1084 tablet_service.cc:1467] Processing CreateTablet for tablet 4a1d8332ecef44829642358719004d99 (DEFAULT_TABLE table=default.Table_2c307103a2e8474b9c7e18b5691f61d4 [id=c65db2976e214ac2b96b15b7a49487c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:31.604964 1990 log.cc:1198] T 25e85b0c5c9a4a3ca35a001359f21167 P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/25e85b0c5c9a4a3ca35a001359f21167
I20250114 02:54:31.605500 1084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a1d8332ecef44829642358719004d99. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:31.605686 1990 ts_tablet_manager.cc:1939] T 25e85b0c5c9a4a3ca35a001359f21167 P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
I20250114 02:54:31.607168 2151 raft_consensus.cc:491] T f40c66314342453b898da3ac0e7e86bf P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:31.607662 1990 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 313) took 1990 ms. Trace:
I20250114 02:54:31.607605 2151 raft_consensus.cc:513] T f40c66314342453b898da3ac0e7e86bf P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:31.609432 2151 leader_election.cc:290] T f40c66314342453b898da3ac0e7e86bf P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:31.602530 1206 tablet_bootstrap.cc:492] T d2a7b2bee5c1440fb705b172c18a9247 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:31.610833 1103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40c66314342453b898da3ac0e7e86bf" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:31.611119 959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40c66314342453b898da3ac0e7e86bf" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
W20250114 02:54:31.612286 753 leader_election.cc:343] T f40c66314342453b898da3ac0e7e86bf P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
W20250114 02:54:31.617372 751 leader_election.cc:343] T f40c66314342453b898da3ac0e7e86bf P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:31.618301 751 leader_election.cc:304] T f40c66314342453b898da3ac0e7e86bf P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:31.619374 2151 raft_consensus.cc:2743] T f40c66314342453b898da3ac0e7e86bf P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:31.620803 1207 ts_tablet_manager.cc:1428] T df045d80d8ce4ff194a6d2039a44bce6 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.108s user 0.008s sys 0.001s
I20250114 02:54:31.607955 1990 rpcz_store.cc:276] 0114 02:54:29.617366 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:29.617475 (+ 109us) service_pool.cc:225] Handling call
0114 02:54:31.478241 (+1860766us) ts_tablet_manager.cc:1145] Deleting tablet 25e85b0c5c9a4a3ca35a001359f21167
0114 02:54:31.478280 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:31.552875 (+ 74595us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:31.578727 (+ 25852us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:31.607606 (+ 28879us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1860247}
I20250114 02:54:31.633692 1990 tablet_replica.cc:331] T 84738a6c78b24c1fb64013048763b18f P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:31.633855 1207 tablet_bootstrap.cc:492] T 719188f639724851a5fc2b37cb60fdbb P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:31.634644 1965 raft_consensus.cc:2238] T e15e6e52d7294aeeb4463b3a76022ec3 P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Raft consensus shutting down.
I20250114 02:54:31.635743 1965 raft_consensus.cc:2267] T e15e6e52d7294aeeb4463b3a76022ec3 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:31.636642 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet 25e85b0c5c9a4a3ca35a001359f21167 (table default.Table_bcf55148b6bc413e8d4e8e31996ff32b [id=d6c16b93fcf5427f9876082d1996a31f]) successfully deleted
I20250114 02:54:31.639467 471 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet 232d6a697f744e179a1a8da492a7d227
I20250114 02:54:31.642627 1075 tablet_service.cc:1514] Processing DeleteTablet for tablet 232d6a697f744e179a1a8da492a7d227 with delete_type TABLET_DATA_DELETED (replaced by tablet 6d5da43a43314707bcdfde1f01efd3dc) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:31.645318 471 catalog_manager.cc:6596] ReplaceTablet: tablet 232d6a697f744e179a1a8da492a7d227 deleted and replaced by tablet 6d5da43a43314707bcdfde1f01efd3dc
I20250114 02:54:31.645586 784 tablet_service.cc:1514] Processing DeleteTablet for tablet 232d6a697f744e179a1a8da492a7d227 with delete_type TABLET_DATA_DELETED (replaced by tablet 6d5da43a43314707bcdfde1f01efd3dc) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:31.650645 2097 raft_consensus.cc:491] T ed7f575653c848d7b076c17c38c4a244 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:31.651281 2097 raft_consensus.cc:513] T ed7f575653c848d7b076c17c38c4a244 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:31.653823 2097 leader_election.cc:290] T ed7f575653c848d7b076c17c38c4a244 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
W20250114 02:54:31.656561 898 leader_election.cc:343] T ed7f575653c848d7b076c17c38c4a244 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Not found: Tablet not found: ed7f575653c848d7b076c17c38c4a244
I20250114 02:54:31.658450 1990 raft_consensus.cc:2238] T 84738a6c78b24c1fb64013048763b18f P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Raft consensus shutting down.
I20250114 02:54:31.659323 1990 raft_consensus.cc:2267] T 84738a6c78b24c1fb64013048763b18f P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:31.661264 1990 ts_tablet_manager.cc:1905] T 84738a6c78b24c1fb64013048763b18f P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:31.662622 932 tablet_service.cc:1514] Processing DeleteTablet for tablet 232d6a697f744e179a1a8da492a7d227 with delete_type TABLET_DATA_DELETED (replaced by tablet 6d5da43a43314707bcdfde1f01efd3dc) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:31.673319 1205 tablet_bootstrap.cc:492] T fc406eab968745d893f1a8538ff61f37 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:31.673528 470 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet 9bfd8fd6d2104fb4a6484cb8c98981bd (table default.Table_0139fab1a81440309d9d7729c3e76c8a [id=2688bb1e2f4f4765986f738256d17f78]). Expected version 1 got 0
I20250114 02:54:31.674206 1207 tablet_bootstrap.cc:654] T 719188f639724851a5fc2b37cb60fdbb P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:31.677470 1075 tablet_service.cc:1467] Processing CreateTablet for tablet dc9329b2184941e88bc324ebc7d816a9 (DEFAULT_TABLE table=default.Table_ceb42359708442c294b3f50aa6eadcc6 [id=0007a4e82dd44243a0db8a29a16350b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:31.687150 1205 ts_tablet_manager.cc:1397] T fc406eab968745d893f1a8538ff61f37 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.181s user 0.011s sys 0.001s
I20250114 02:54:31.687976 1965 ts_tablet_manager.cc:1905] T e15e6e52d7294aeeb4463b3a76022ec3 P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:31.690057 1205 raft_consensus.cc:357] T fc406eab968745d893f1a8538ff61f37 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:31.690966 1205 raft_consensus.cc:383] T fc406eab968745d893f1a8538ff61f37 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:31.691278 1205 raft_consensus.cc:738] T fc406eab968745d893f1a8538ff61f37 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:31.681243 932 tablet_service.cc:1467] Processing CreateTablet for tablet dc9329b2184941e88bc324ebc7d816a9 (DEFAULT_TABLE table=default.Table_ceb42359708442c294b3f50aa6eadcc6 [id=0007a4e82dd44243a0db8a29a16350b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:31.692824 932 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc9329b2184941e88bc324ebc7d816a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:31.691128 471 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_c8f09aee14204d5ea1c506f230fb81b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:31.698567 471 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c8f09aee14204d5ea1c506f230fb81b6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:31.701582 470 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet 044832851f804208b0a371eb0857dbf3 (table default.Table_0518d8bed7304c86aff32e05e6f07c0d [id=515a2fe3f1e84e82bccc02ef5cecef1b]). Expected version 1 got 0
I20250114 02:54:31.701649 1102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7f575653c848d7b076c17c38c4a244" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
W20250114 02:54:31.703657 898 leader_election.cc:343] T ed7f575653c848d7b076c17c38c4a244 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:31.704157 898 leader_election.cc:304] T ed7f575653c848d7b076c17c38c4a244 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 525198593c8e47afaee54e74267b87e3; no voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:31.705246 2097 raft_consensus.cc:2743] T ed7f575653c848d7b076c17c38c4a244 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:31.709517 784 tablet_service.cc:1467] Processing CreateTablet for tablet dc9329b2184941e88bc324ebc7d816a9 (DEFAULT_TABLE table=default.Table_ceb42359708442c294b3f50aa6eadcc6 [id=0007a4e82dd44243a0db8a29a16350b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:31.710786 784 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc9329b2184941e88bc324ebc7d816a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:31.716020 931 tablet_service.cc:1467] Processing CreateTablet for tablet 5166e79dac1646b1b9fac6c3184e3139 (DEFAULT_TABLE table=default.Table_dda4f31205ab46c7a3af70bbb8037ec8 [id=94052a8a70044fab9b2753c326ad23d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:31.716416 1992 ts_tablet_manager.cc:1918] T e15e6e52d7294aeeb4463b3a76022ec3 P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:31.716759 1992 log.cc:1198] T e15e6e52d7294aeeb4463b3a76022ec3 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/e15e6e52d7294aeeb4463b3a76022ec3
I20250114 02:54:31.717424 931 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5166e79dac1646b1b9fac6c3184e3139. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:31.718696 470 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet 92cd8eccacf34ab7a66b3bbc69df3697 (table default.Table_990b8d5f1bd146d6b358457f954780c9 [id=26627ccdc808451fb9593ad7a6df6991]). Expected version 2 got 0
I20250114 02:54:31.719043 785 tablet_service.cc:1467] Processing CreateTablet for tablet 5166e79dac1646b1b9fac6c3184e3139 (DEFAULT_TABLE table=default.Table_dda4f31205ab46c7a3af70bbb8037ec8 [id=94052a8a70044fab9b2753c326ad23d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:31.720399 785 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5166e79dac1646b1b9fac6c3184e3139. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:31.720690 470 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet edc69b66bea54625a4b15867b75986b7 (table default.Table_0a0bda55a4a9476c98d961822b8a19bd [id=fb2be5c1ca9444a197f3315ea9c59b09]). Expected version 1 got 0
I20250114 02:54:31.728807 1992 ts_tablet_manager.cc:1939] T e15e6e52d7294aeeb4463b3a76022ec3 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:31.730713 1992 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 275) took 2309 ms. Trace:
I20250114 02:54:31.731010 1992 rpcz_store.cc:276] 0114 02:54:29.421082 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:29.427847 (+ 6765us) service_pool.cc:225] Handling call
0114 02:54:31.338184 (+1910337us) ts_tablet_manager.cc:1145] Deleting tablet e15e6e52d7294aeeb4463b3a76022ec3
0114 02:54:31.338457 (+ 273us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:31.501501 (+163044us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:31.692143 (+190642us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:31.730658 (+ 38515us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1908632}
I20250114 02:54:31.732638 1992 tablet_replica.cc:331] T 6e858a8449444ef999b9bbef612249d3 P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:31.732748 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet e15e6e52d7294aeeb4463b3a76022ec3 (table default.Table_d1de82caa5b94791b6c7dfed1c376e8a [id=03c775a54e9e443cbc4fab853bd6d49c]) successfully deleted
I20250114 02:54:31.733363 1992 raft_consensus.cc:2238] T 6e858a8449444ef999b9bbef612249d3 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:31.733780 1992 raft_consensus.cc:2267] T 6e858a8449444ef999b9bbef612249d3 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:31.735430 1992 ts_tablet_manager.cc:1905] T 6e858a8449444ef999b9bbef612249d3 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:31.739090 471 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_7416825652064407ab1bc5469bc21167"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:31.740694 471 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7416825652064407ab1bc5469bc21167 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:31.743786 934 tablet_service.cc:1467] Processing CreateTablet for tablet af3422618f89492e8185fcce8883cfe7 (DEFAULT_TABLE table=default.Table_17190cd416734d35abcc8c6b354e5bb1 [id=3b8c1a8e39d84aef81cf505719161711]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:31.744853 934 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af3422618f89492e8185fcce8883cfe7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:31.746953 2151 raft_consensus.cc:491] T 967f4d5b944543d49e87645d069b5b55 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:31.747542 1990 ts_tablet_manager.cc:1918] T 84738a6c78b24c1fb64013048763b18f P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:31.747395 2151 raft_consensus.cc:513] T 967f4d5b944543d49e87645d069b5b55 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:31.747871 1990 log.cc:1198] T 84738a6c78b24c1fb64013048763b18f P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/84738a6c78b24c1fb64013048763b18f
I20250114 02:54:31.749310 2151 leader_election.cc:290] T 967f4d5b944543d49e87645d069b5b55 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:31.758209 794 tablet_service.cc:1467] Processing CreateTablet for tablet af3422618f89492e8185fcce8883cfe7 (DEFAULT_TABLE table=default.Table_17190cd416734d35abcc8c6b354e5bb1 [id=3b8c1a8e39d84aef81cf505719161711]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:31.759230 794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af3422618f89492e8185fcce8883cfe7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:31.762883 470 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet 15704785f56e43a3af69f181c75877ce (table default.Table_9da872c9a22e4d07947df8db0524e61f [id=1f964e9ed159449580fe5326673c18ab]). Expected version 1 got 0
I20250114 02:54:31.763983 1084 tablet_service.cc:1467] Processing CreateTablet for tablet 5166e79dac1646b1b9fac6c3184e3139 (DEFAULT_TABLE table=default.Table_dda4f31205ab46c7a3af70bbb8037ec8 [id=94052a8a70044fab9b2753c326ad23d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:31.765324 787 tablet_service.cc:1467] Processing CreateTablet for tablet 53929328fa51408aa98d7825018b6fdb (DEFAULT_TABLE table=default.Table_35a78676616e4b20906b262c9d0c0d9e [id=e984149e9da444f1b2802e3f35a60fd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:31.766309 787 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53929328fa51408aa98d7825018b6fdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:31.768579 959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "967f4d5b944543d49e87645d069b5b55" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:31.769214 1102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "967f4d5b944543d49e87645d069b5b55" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
W20250114 02:54:31.769675 751 leader_election.cc:343] T 967f4d5b944543d49e87645d069b5b55 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
W20250114 02:54:31.773123 753 leader_election.cc:343] T 967f4d5b944543d49e87645d069b5b55 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:31.773625 753 leader_election.cc:304] T 967f4d5b944543d49e87645d069b5b55 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:31.775442 2151 raft_consensus.cc:2743] T 967f4d5b944543d49e87645d069b5b55 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:31.776600 935 tablet_service.cc:1467] Processing CreateTablet for tablet 53929328fa51408aa98d7825018b6fdb (DEFAULT_TABLE table=default.Table_35a78676616e4b20906b262c9d0c0d9e [id=e984149e9da444f1b2802e3f35a60fd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:31.777634 935 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53929328fa51408aa98d7825018b6fdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:31.778340 1086 tablet_service.cc:1467] Processing CreateTablet for tablet af3422618f89492e8185fcce8883cfe7 (DEFAULT_TABLE table=default.Table_17190cd416734d35abcc8c6b354e5bb1 [id=3b8c1a8e39d84aef81cf505719161711]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:31.785660 1075 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc9329b2184941e88bc324ebc7d816a9. 1 dirs total, 0 dirs full, 0 dirs failed
W20250114 02:54:31.757157 1041 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.0.107.126:45367, user_credentials={real_user=slave}} blocked reactor thread for 56144.5us
I20250114 02:54:31.794025 1086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af3422618f89492e8185fcce8883cfe7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:31.812453 1084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5166e79dac1646b1b9fac6c3184e3139. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:31.764721 1990 ts_tablet_manager.cc:1939] T 84738a6c78b24c1fb64013048763b18f P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
I20250114 02:54:31.819579 1990 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 314) took 2058 ms. Trace:
I20250114 02:54:31.819877 1990 rpcz_store.cc:276] 0114 02:54:29.761122 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:29.761241 (+ 119us) service_pool.cc:225] Handling call
0114 02:54:31.633533 (+1872292us) ts_tablet_manager.cc:1145] Deleting tablet 84738a6c78b24c1fb64013048763b18f
0114 02:54:31.633585 (+ 52us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:31.671947 (+ 38362us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:31.747325 (+ 75378us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:31.819522 (+ 72197us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23903,"tablet-delete.queue_time_us":1871783}
I20250114 02:54:31.821007 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet 84738a6c78b24c1fb64013048763b18f (table default.Table_7058838b6bcb427786778096cf6f52b3 [id=b5d3461f83da46faaa772376a318b6b6]) successfully deleted
I20250114 02:54:31.821077 1990 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 316) took 2042 ms. Trace:
I20250114 02:54:31.821336 1990 rpcz_store.cc:276] 0114 02:54:29.778225 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:29.778355 (+ 130us) service_pool.cc:225] Handling call
0114 02:54:31.820848 (+2042493us) ts_tablet_manager.cc:1145] Deleting tablet 1381e26725174b9c877075cf5444f566
0114 02:54:31.820883 (+ 35us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:31.821026 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2041929}
W20250114 02:54:31.822292 458 catalog_manager.cc:4839] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 1381e26725174b9c877075cf5444f566 because the tablet was not found. No further retry: Not found: Tablet not found: 1381e26725174b9c877075cf5444f566
I20250114 02:54:31.822374 1990 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 322) took 1665 ms. Trace:
I20250114 02:54:31.821497 1205 consensus_queue.cc:260] T fc406eab968745d893f1a8538ff61f37 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:31.822620 1990 rpcz_store.cc:276] 0114 02:54:30.156679 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:30.156809 (+ 130us) service_pool.cc:225] Handling call
0114 02:54:31.822077 (+1665268us) ts_tablet_manager.cc:1145] Deleting tablet 4ab79e0a576843e89a2d524608c5b7f4
0114 02:54:31.822109 (+ 32us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:31.822331 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1662715}
I20250114 02:54:31.823602 1990 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 323) took 1661 ms. Trace:
I20250114 02:54:31.827576 1205 ts_tablet_manager.cc:1428] T fc406eab968745d893f1a8538ff61f37 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.140s user 0.006s sys 0.000s
W20250114 02:54:31.829033 458 catalog_manager.cc:4852] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 4ab79e0a576843e89a2d524608c5b7f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4ab79e0a576843e89a2d524608c5b7f4 already in progress: creating tablet
I20250114 02:54:31.851693 1076 tablet_service.cc:1467] Processing CreateTablet for tablet 53929328fa51408aa98d7825018b6fdb (DEFAULT_TABLE table=default.Table_35a78676616e4b20906b262c9d0c0d9e [id=e984149e9da444f1b2802e3f35a60fd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:31.852002 2275 raft_consensus.cc:491] T c86562688d5b4fffb71cbdf7be0afc37 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:31.852850 1076 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53929328fa51408aa98d7825018b6fdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:31.852420 2275 raft_consensus.cc:513] T c86562688d5b4fffb71cbdf7be0afc37 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:31.854714 2275 leader_election.cc:290] T c86562688d5b4fffb71cbdf7be0afc37 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:31.855975 959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c86562688d5b4fffb71cbdf7be0afc37" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:31.856679 959 raft_consensus.cc:2388] T c86562688d5b4fffb71cbdf7be0afc37 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 90d6c2c58bb54e0eafb365625ac33871 in current term 1: Already voted for candidate 525198593c8e47afaee54e74267b87e3 in this term.
I20250114 02:54:31.867072 813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c86562688d5b4fffb71cbdf7be0afc37" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:31.867964 813 raft_consensus.cc:2388] T c86562688d5b4fffb71cbdf7be0afc37 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 90d6c2c58bb54e0eafb365625ac33871 in current term 1: Already voted for candidate 819f9e3412e54d44abb11d8b3ca877c2 in this term.
I20250114 02:54:31.870333 2097 raft_consensus.cc:491] T 59a46bc865cc48df8e98598764fd1b54 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:31.870824 2097 raft_consensus.cc:513] T 59a46bc865cc48df8e98598764fd1b54 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:31.872532 2097 leader_election.cc:290] T 59a46bc865cc48df8e98598764fd1b54 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
W20250114 02:54:31.894098 898 leader_election.cc:343] T 59a46bc865cc48df8e98598764fd1b54 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Not found: Tablet not found: 59a46bc865cc48df8e98598764fd1b54
I20250114 02:54:31.896523 466 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet edc69b66bea54625a4b15867b75986b7 (table default.Table_0a0bda55a4a9476c98d961822b8a19bd [id=fb2be5c1ca9444a197f3315ea9c59b09]). Expected version 1 got 0
I20250114 02:54:31.897255 466 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 38fe8829d98547fb907ef78099df3d7f (table default.Table_60450873174a4c80aa7e2122343d82be [id=9299683349354f879847b7c9d93f6072]). Expected version 1 got 0
I20250114 02:54:31.905102 1205 tablet_bootstrap.cc:492] T 97f4e061d3b94ee9982410b7ce135453 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:31.906476 466 catalog_manager.cc:5526] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 819f9e3412e54d44abb11d8b3ca877c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67). New cstate: current_term: 1 leader_uuid: "90d6c2c58bb54e0eafb365625ac33871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } }
I20250114 02:54:31.908938 1042 leader_election.cc:304] T c86562688d5b4fffb71cbdf7be0afc37 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90d6c2c58bb54e0eafb365625ac33871; no voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2
I20250114 02:54:31.911219 1102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59a46bc865cc48df8e98598764fd1b54" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
W20250114 02:54:31.912976 898 leader_election.cc:343] T 59a46bc865cc48df8e98598764fd1b54 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:31.913460 898 leader_election.cc:304] T 59a46bc865cc48df8e98598764fd1b54 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 525198593c8e47afaee54e74267b87e3; no voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:31.920416 2273 raft_consensus.cc:3054] T c86562688d5b4fffb71cbdf7be0afc37 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:31.921564 2097 raft_consensus.cc:2743] T 59a46bc865cc48df8e98598764fd1b54 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:31.823854 1990 rpcz_store.cc:276] 0114 02:54:30.161877 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:30.162859 (+ 982us) service_pool.cc:225] Handling call
0114 02:54:31.823355 (+1660496us) ts_tablet_manager.cc:1145] Deleting tablet 3161f75bc4e84000af022faa9804c810
0114 02:54:31.823384 (+ 29us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:31.823561 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1659880}
I20250114 02:54:31.953596 1207 tablet_bootstrap.cc:492] T 719188f639724851a5fc2b37cb60fdbb P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:31.954102 1207 ts_tablet_manager.cc:1397] T 719188f639724851a5fc2b37cb60fdbb P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.320s user 0.010s sys 0.003s
W20250114 02:54:31.932940 458 catalog_manager.cc:4852] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 3161f75bc4e84000af022faa9804c810 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3161f75bc4e84000af022faa9804c810 already in progress: creating tablet
I20250114 02:54:31.960219 466 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 92cd8eccacf34ab7a66b3bbc69df3697 (table default.Table_990b8d5f1bd146d6b358457f954780c9 [id=26627ccdc808451fb9593ad7a6df6991]). Expected version 2 got 0
I20250114 02:54:31.962019 1965 ts_tablet_manager.cc:1918] T e15e6e52d7294aeeb4463b3a76022ec3 P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:31.942999 471 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_5623f4fcb2fb42ef8cf628812939a998"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:31.964679 1990 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 324) took 1667 ms. Trace:
I20250114 02:54:31.965040 1990 rpcz_store.cc:276] 0114 02:54:30.297263 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:30.312355 (+ 15092us) service_pool.cc:225] Handling call
0114 02:54:31.933446 (+1621091us) ts_tablet_manager.cc:1145] Deleting tablet 25bce689024f49a49496e415efe91fdc
0114 02:54:31.933489 (+ 43us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:31.964613 (+ 31124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1620461}
W20250114 02:54:31.965955 471 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5623f4fcb2fb42ef8cf628812939a998 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250114 02:54:31.966249 458 catalog_manager.cc:4852] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 25bce689024f49a49496e415efe91fdc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 25bce689024f49a49496e415efe91fdc already in progress: creating tablet
I20250114 02:54:31.967295 1990 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 326) took 1040 ms. Trace:
I20250114 02:54:31.967666 1990 rpcz_store.cc:276] 0114 02:54:30.926683 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:30.928658 (+ 1975us) service_pool.cc:225] Handling call
0114 02:54:31.966967 (+1038309us) ts_tablet_manager.cc:1145] Deleting tablet 99b940d872ce4781907517c40fe08781
0114 02:54:31.967003 (+ 36us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:31.967229 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1036211}
I20250114 02:54:31.968341 466 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet a52d6749ff834b62a8e20a2f45935240 (table default.Table_1c9b7896941e4289b1c12e20c210c2c1 [id=b3ee15b9f10f4d489edbde3af9034d25]). Expected version 1 got 0
I20250114 02:54:31.968812 1205 tablet_bootstrap.cc:654] T 97f4e061d3b94ee9982410b7ce135453 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:31.968834 1965 log.cc:1198] T e15e6e52d7294aeeb4463b3a76022ec3 P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/e15e6e52d7294aeeb4463b3a76022ec3
I20250114 02:54:31.971329 1965 ts_tablet_manager.cc:1939] T e15e6e52d7294aeeb4463b3a76022ec3 P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
W20250114 02:54:31.969012 458 catalog_manager.cc:4852] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 99b940d872ce4781907517c40fe08781 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 99b940d872ce4781907517c40fe08781 already in progress: creating tablet
I20250114 02:54:31.972671 1207 raft_consensus.cc:357] T 719188f639724851a5fc2b37cb60fdbb P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:31.973510 1207 raft_consensus.cc:383] T 719188f639724851a5fc2b37cb60fdbb P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:31.974041 1965 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55068 (request call id 317) took 2508 ms. Trace:
I20250114 02:54:31.984395 1965 rpcz_store.cc:276] 0114 02:54:29.465963 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:29.466091 (+ 128us) service_pool.cc:225] Handling call
0114 02:54:31.551512 (+2085421us) ts_tablet_manager.cc:1145] Deleting tablet e15e6e52d7294aeeb4463b3a76022ec3
0114 02:54:31.551547 (+ 35us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:31.744689 (+193142us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:31.961793 (+217104us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:31.973989 (+ 12196us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2084842}
I20250114 02:54:31.986105 1965 tablet_replica.cc:331] T 25e85b0c5c9a4a3ca35a001359f21167 P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:31.986829 1965 raft_consensus.cc:2238] T 25e85b0c5c9a4a3ca35a001359f21167 P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Raft consensus shutting down.
I20250114 02:54:31.987673 1965 raft_consensus.cc:2267] T 25e85b0c5c9a4a3ca35a001359f21167 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250114 02:54:31.990723 458 catalog_manager.cc:4852] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 41814f82964a4bcb8dbb6d3738052448 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 41814f82964a4bcb8dbb6d3738052448 already in progress: creating tablet
W20250114 02:54:31.992179 458 catalog_manager.cc:4852] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 587a5f39282a40d0a8c1591fdf36f341 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 587a5f39282a40d0a8c1591fdf36f341 already in progress: creating tablet
I20250114 02:54:31.994844 1207 raft_consensus.cc:738] T 719188f639724851a5fc2b37cb60fdbb P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:31.995620 1207 consensus_queue.cc:260] T 719188f639724851a5fc2b37cb60fdbb P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:31.995972 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet e15e6e52d7294aeeb4463b3a76022ec3 (table default.Table_d1de82caa5b94791b6c7dfed1c376e8a [id=03c775a54e9e443cbc4fab853bd6d49c]) successfully deleted
I20250114 02:54:31.989280 2273 raft_consensus.cc:2743] T c86562688d5b4fffb71cbdf7be0afc37 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:31.992944 1990 tablet_replica.cc:331] T 2837990edc4d4bea871a8114533c9352 P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:31.975584 466 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 719188f639724851a5fc2b37cb60fdbb (table default.Table_8f8a18d7868c44bbb4d66a0692b08186 [id=e6292728826f437cb75c524a8aef2e87]). Expected version 1 got 0
I20250114 02:54:32.014978 1965 ts_tablet_manager.cc:1905] T 25e85b0c5c9a4a3ca35a001359f21167 P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:32.058547 1965 ts_tablet_manager.cc:1918] T 25e85b0c5c9a4a3ca35a001359f21167 P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:32.059017 1965 log.cc:1198] T 25e85b0c5c9a4a3ca35a001359f21167 P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/25e85b0c5c9a4a3ca35a001359f21167
I20250114 02:54:32.059778 1965 ts_tablet_manager.cc:1939] T 25e85b0c5c9a4a3ca35a001359f21167 P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:32.061834 1965 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55068 (request call id 318) took 2500 ms. Trace:
I20250114 02:54:32.062196 1965 rpcz_store.cc:276] 0114 02:54:29.561022 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:29.561156 (+ 134us) service_pool.cc:225] Handling call
0114 02:54:31.985964 (+2424808us) ts_tablet_manager.cc:1145] Deleting tablet 25e85b0c5c9a4a3ca35a001359f21167
0114 02:54:31.986003 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:32.038727 (+ 52724us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:32.058322 (+ 19595us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:32.061775 (+ 3453us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2424234}
I20250114 02:54:32.064738 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet 25e85b0c5c9a4a3ca35a001359f21167 (table default.Table_bcf55148b6bc413e8d4e8e31996ff32b [id=d6c16b93fcf5427f9876082d1996a31f]) successfully deleted
I20250114 02:54:32.065810 1965 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55068 (request call id 323) took 1920 ms. Trace:
I20250114 02:54:32.066145 1965 rpcz_store.cc:276] 0114 02:54:30.145029 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:30.145166 (+ 137us) service_pool.cc:225] Handling call
0114 02:54:32.065477 (+1920311us) ts_tablet_manager.cc:1145] Deleting tablet 4ab79e0a576843e89a2d524608c5b7f4
0114 02:54:32.065514 (+ 37us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:32.065762 (+ 248us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1919714}
I20250114 02:54:32.068588 1990 raft_consensus.cc:2238] T 2837990edc4d4bea871a8114533c9352 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Raft consensus shutting down.
W20250114 02:54:32.072328 456 catalog_manager.cc:4852] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet 4ab79e0a576843e89a2d524608c5b7f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4ab79e0a576843e89a2d524608c5b7f4 already in progress: creating tablet
I20250114 02:54:32.073414 1965 tablet_replica.cc:331] T 3161f75bc4e84000af022faa9804c810 P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:32.074143 1965 raft_consensus.cc:2238] T 3161f75bc4e84000af022faa9804c810 P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Raft consensus shutting down.
I20250114 02:54:32.074821 1965 pending_rounds.cc:62] T 3161f75bc4e84000af022faa9804c810 P 525198593c8e47afaee54e74267b87e3: Trying to abort 1 pending ops.
I20250114 02:54:32.078455 1207 ts_tablet_manager.cc:1428] T 719188f639724851a5fc2b37cb60fdbb P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.124s user 0.007s sys 0.000s
I20250114 02:54:32.080081 1207 tablet_bootstrap.cc:492] T 97f4e061d3b94ee9982410b7ce135453 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:32.075079 1965 pending_rounds.cc:69] T 3161f75bc4e84000af022faa9804c810 P 525198593c8e47afaee54e74267b87e3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114028120141697024 op_type: NO_OP noop_request { }
I20250114 02:54:32.097724 1965 raft_consensus.cc:2883] T 3161f75bc4e84000af022faa9804c810 P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250114 02:54:32.098268 1990 raft_consensus.cc:2267] T 2837990edc4d4bea871a8114533c9352 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:32.098728 1965 raft_consensus.cc:2267] T 3161f75bc4e84000af022faa9804c810 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:32.105862 1205 tablet_bootstrap.cc:492] T 97f4e061d3b94ee9982410b7ce135453 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:32.106146 1965 ts_tablet_manager.cc:1905] T 3161f75bc4e84000af022faa9804c810 P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:32.106941 1990 ts_tablet_manager.cc:1905] T 2837990edc4d4bea871a8114533c9352 P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:32.106262 1205 ts_tablet_manager.cc:1397] T 97f4e061d3b94ee9982410b7ce135453 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.201s user 0.017s sys 0.008s
I20250114 02:54:32.123163 1205 raft_consensus.cc:357] T 97f4e061d3b94ee9982410b7ce135453 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:32.133280 1965 ts_tablet_manager.cc:1918] T 3161f75bc4e84000af022faa9804c810 P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:32.139689 2308 raft_consensus.cc:491] T 30a56fb2d941470590e17f37b90aef26 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:32.140379 1965 log.cc:1198] T 3161f75bc4e84000af022faa9804c810 P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/3161f75bc4e84000af022faa9804c810
I20250114 02:54:32.141243 1965 ts_tablet_manager.cc:1939] T 3161f75bc4e84000af022faa9804c810 P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:32.143302 1965 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55068 (request call id 324) took 1982 ms. Trace:
I20250114 02:54:32.143719 1965 rpcz_store.cc:276] 0114 02:54:30.160573 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:30.160741 (+ 168us) service_pool.cc:225] Handling call
0114 02:54:32.073264 (+1912523us) ts_tablet_manager.cc:1145] Deleting tablet 3161f75bc4e84000af022faa9804c810
0114 02:54:32.073312 (+ 48us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:32.125978 (+ 52666us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:32.133090 (+ 7112us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:32.143249 (+ 10159us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1911958}
I20250114 02:54:32.144943 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet 3161f75bc4e84000af022faa9804c810 (table default.Table_35a78676616e4b20906b262c9d0c0d9e [id=e984149e9da444f1b2802e3f35a60fd6]) successfully deleted
I20250114 02:54:32.145423 1965 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55068 (request call id 325) took 1910 ms. Trace:
I20250114 02:54:32.145762 1965 rpcz_store.cc:276] 0114 02:54:30.234823 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:30.234976 (+ 153us) service_pool.cc:225] Handling call
0114 02:54:32.145059 (+1910083us) ts_tablet_manager.cc:1145] Deleting tablet 25bce689024f49a49496e415efe91fdc
0114 02:54:32.145119 (+ 60us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:32.145376 (+ 257us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1909456}
W20250114 02:54:32.146863 456 catalog_manager.cc:4852] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet 25bce689024f49a49496e415efe91fdc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 25bce689024f49a49496e415efe91fdc already in progress: creating tablet
I20250114 02:54:32.147722 1965 tablet_replica.cc:331] T 99b940d872ce4781907517c40fe08781 P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:32.146930 2308 raft_consensus.cc:513] T 30a56fb2d941470590e17f37b90aef26 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:32.151757 1965 raft_consensus.cc:2238] T 99b940d872ce4781907517c40fe08781 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:32.152288 1965 raft_consensus.cc:2267] T 99b940d872ce4781907517c40fe08781 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:32.154022 1965 ts_tablet_manager.cc:1905] T 99b940d872ce4781907517c40fe08781 P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:32.156075 2308 leader_election.cc:290] T 30a56fb2d941470590e17f37b90aef26 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:32.181970 1965 ts_tablet_manager.cc:1918] T 99b940d872ce4781907517c40fe08781 P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250114 02:54:32.182376 1965 log.cc:1198] T 99b940d872ce4781907517c40fe08781 P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/99b940d872ce4781907517c40fe08781
I20250114 02:54:32.182816 1990 ts_tablet_manager.cc:1918] T 2837990edc4d4bea871a8114533c9352 P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20250114 02:54:32.183094 1965 ts_tablet_manager.cc:1939] T 99b940d872ce4781907517c40fe08781 P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:32.183218 1990 log.cc:1198] T 2837990edc4d4bea871a8114533c9352 P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/2837990edc4d4bea871a8114533c9352
I20250114 02:54:32.183952 1990 ts_tablet_manager.cc:1939] T 2837990edc4d4bea871a8114533c9352 P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
I20250114 02:54:32.191282 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet 2837990edc4d4bea871a8114533c9352 (table default.Table_ba8866c1c90041268e09ff59a46503f1 [id=0b1b2c3855f54d5c97bd2866dfbd8a5d]) successfully deleted
I20250114 02:54:32.195534 1965 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55068 (request call id 329) took 1278 ms. Trace:
I20250114 02:54:32.197546 1965 rpcz_store.cc:276] 0114 02:54:30.916973 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:30.952396 (+ 35423us) service_pool.cc:225] Handling call
0114 02:54:32.147561 (+1195165us) ts_tablet_manager.cc:1145] Deleting tablet 99b940d872ce4781907517c40fe08781
0114 02:54:32.147616 (+ 55us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:32.174680 (+ 27064us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:32.181772 (+ 7092us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:32.195461 (+ 13689us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1182571}
I20250114 02:54:32.205793 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet 99b940d872ce4781907517c40fe08781 (table default.Table_0943b1ca347042c9a6043006bc566701 [id=e3418787ef0a4e22979fb41d4f94ebb0]) successfully deleted
I20250114 02:54:32.207091 1965 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55068 (request call id 330) took 1170 ms. Trace:
I20250114 02:54:32.207430 1965 rpcz_store.cc:276] 0114 02:54:31.036450 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:31.036562 (+ 112us) service_pool.cc:225] Handling call
0114 02:54:32.206735 (+1170173us) ts_tablet_manager.cc:1145] Deleting tablet 41814f82964a4bcb8dbb6d3738052448
0114 02:54:32.206780 (+ 45us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:32.207037 (+ 257us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1169501}
W20250114 02:54:32.208645 456 catalog_manager.cc:4852] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet 41814f82964a4bcb8dbb6d3738052448 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 41814f82964a4bcb8dbb6d3738052448 already in progress: creating tablet
I20250114 02:54:32.209671 1965 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55068 (request call id 331) took 1161 ms. Trace:
I20250114 02:54:32.209966 1965 rpcz_store.cc:276] 0114 02:54:31.048432 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:31.048565 (+ 133us) service_pool.cc:225] Handling call
0114 02:54:32.209320 (+1160755us) ts_tablet_manager.cc:1145] Deleting tablet 587a5f39282a40d0a8c1591fdf36f341
0114 02:54:32.209358 (+ 38us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:32.209625 (+ 267us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1160167}
W20250114 02:54:32.211064 456 catalog_manager.cc:4852] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet 587a5f39282a40d0a8c1591fdf36f341 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 587a5f39282a40d0a8c1591fdf36f341 already in progress: creating tablet
I20250114 02:54:32.211943 1965 tablet_replica.cc:331] T 2837990edc4d4bea871a8114533c9352 P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:32.212956 1965 raft_consensus.cc:2238] T 2837990edc4d4bea871a8114533c9352 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:32.213544 1965 raft_consensus.cc:2267] T 2837990edc4d4bea871a8114533c9352 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:32.215549 1965 ts_tablet_manager.cc:1905] T 2837990edc4d4bea871a8114533c9352 P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:32.221257 1206 tablet_bootstrap.cc:654] T d2a7b2bee5c1440fb705b172c18a9247 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:32.223344 1992 ts_tablet_manager.cc:1918] T 6e858a8449444ef999b9bbef612249d3 P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:32.224023 1992 log.cc:1198] T 6e858a8449444ef999b9bbef612249d3 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/6e858a8449444ef999b9bbef612249d3
I20250114 02:54:32.225041 1992 ts_tablet_manager.cc:1939] T 6e858a8449444ef999b9bbef612249d3 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:32.186978 1207 tablet_bootstrap.cc:654] T 97f4e061d3b94ee9982410b7ce135453 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:32.230942 2273 raft_consensus.cc:491] T 719188f639724851a5fc2b37cb60fdbb P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:32.236249 2273 raft_consensus.cc:513] T 719188f639724851a5fc2b37cb60fdbb P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:32.238243 2273 leader_election.cc:290] T 719188f639724851a5fc2b37cb60fdbb P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:32.239732 959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "719188f639724851a5fc2b37cb60fdbb" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:32.240415 959 raft_consensus.cc:2388] T 719188f639724851a5fc2b37cb60fdbb P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 90d6c2c58bb54e0eafb365625ac33871 in current term 1: Already voted for candidate 525198593c8e47afaee54e74267b87e3 in this term.
I20250114 02:54:32.192221 1990 tablet_replica.cc:331] T a1cf92e5275b454ca893b12f2d90d25b P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:32.241565 1990 raft_consensus.cc:2238] T a1cf92e5275b454ca893b12f2d90d25b P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:32.242086 1990 raft_consensus.cc:2267] T a1cf92e5275b454ca893b12f2d90d25b P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:32.124051 1205 raft_consensus.cc:383] T 97f4e061d3b94ee9982410b7ce135453 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:32.243533 1205 raft_consensus.cc:738] T 97f4e061d3b94ee9982410b7ce135453 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:32.243888 1990 ts_tablet_manager.cc:1905] T a1cf92e5275b454ca893b12f2d90d25b P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:32.235880 1965 ts_tablet_manager.cc:1918] T 2837990edc4d4bea871a8114533c9352 P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20250114 02:54:32.248541 1965 log.cc:1198] T 2837990edc4d4bea871a8114533c9352 P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/2837990edc4d4bea871a8114533c9352
I20250114 02:54:32.249420 1965 ts_tablet_manager.cc:1939] T 2837990edc4d4bea871a8114533c9352 P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:32.248023 1205 consensus_queue.cc:260] T 97f4e061d3b94ee9982410b7ce135453 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:32.255842 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet 2837990edc4d4bea871a8114533c9352 (table default.Table_ba8866c1c90041268e09ff59a46503f1 [id=0b1b2c3855f54d5c97bd2866dfbd8a5d]) successfully deleted
I20250114 02:54:32.262571 1965 tablet_replica.cc:331] T a1cf92e5275b454ca893b12f2d90d25b P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:32.264699 1990 ts_tablet_manager.cc:1918] T a1cf92e5275b454ca893b12f2d90d25b P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20250114 02:54:32.265022 1990 log.cc:1198] T a1cf92e5275b454ca893b12f2d90d25b P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/a1cf92e5275b454ca893b12f2d90d25b
I20250114 02:54:32.265698 1990 ts_tablet_manager.cc:1939] T a1cf92e5275b454ca893b12f2d90d25b P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
I20250114 02:54:32.267719 1965 raft_consensus.cc:2238] T a1cf92e5275b454ca893b12f2d90d25b P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Raft consensus shutting down.
I20250114 02:54:32.267987 1205 ts_tablet_manager.cc:1428] T 97f4e061d3b94ee9982410b7ce135453 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.148s user 0.006s sys 0.000s
I20250114 02:54:32.268896 1965 raft_consensus.cc:2267] T a1cf92e5275b454ca893b12f2d90d25b P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:32.266469 1992 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 276) took 2797 ms. Trace:
I20250114 02:54:32.269618 2326 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.124:39105 every 8 connection(s)
I20250114 02:54:32.269591 1992 rpcz_store.cc:276] 0114 02:54:29.428933 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:29.431106 (+ 2173us) service_pool.cc:225] Handling call
0114 02:54:31.732506 (+2301400us) ts_tablet_manager.cc:1145] Deleting tablet 6e858a8449444ef999b9bbef612249d3
0114 02:54:31.732542 (+ 36us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:31.963420 (+230878us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:32.222842 (+259422us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:32.226053 (+ 3211us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2270946}
I20250114 02:54:32.270510 1423 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.124:39105
I20250114 02:54:32.270972 1992 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 277) took 2839 ms. Trace:
I20250114 02:54:32.270964 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet 6e858a8449444ef999b9bbef612249d3 (table default.Table_5a9480c6474848d6b400e7dd274d15b0 [id=a17ad30ef32f4cbcb22b5a7d1c7096ff]) successfully deleted
I20250114 02:54:32.271224 1992 rpcz_store.cc:276] 0114 02:54:29.431966 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:29.458645 (+ 26679us) service_pool.cc:225] Handling call
0114 02:54:32.270627 (+2811982us) ts_tablet_manager.cc:1145] Deleting tablet 46360f726fad4534a2daf67581346fc1
0114 02:54:32.270710 (+ 83us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:32.270923 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2733341}
W20250114 02:54:32.272383 458 catalog_manager.cc:4839] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet 46360f726fad4534a2daf67581346fc1 because the tablet was not found. No further retry: Not found: Tablet not found: 46360f726fad4534a2daf67581346fc1
I20250114 02:54:32.272550 1992 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 279) took 2791 ms. Trace:
I20250114 02:54:32.272825 1992 rpcz_store.cc:276] 0114 02:54:29.480860 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:29.486489 (+ 5629us) service_pool.cc:225] Handling call
0114 02:54:32.272002 (+2785513us) ts_tablet_manager.cc:1145] Deleting tablet f0bd9bd5a17d418d8082ed0ebd1bf4cc
0114 02:54:32.272056 (+ 54us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:32.272506 (+ 450us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2698576}
I20250114 02:54:32.273849 1992 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 281) took 2785 ms. Trace:
W20250114 02:54:32.273885 458 catalog_manager.cc:4839] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet f0bd9bd5a17d418d8082ed0ebd1bf4cc because the tablet was not found. No further retry: Not found: Tablet not found: f0bd9bd5a17d418d8082ed0ebd1bf4cc
I20250114 02:54:32.274145 1992 rpcz_store.cc:276] 0114 02:54:29.488039 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:29.488168 (+ 129us) service_pool.cc:225] Handling call
0114 02:54:32.273564 (+2785396us) ts_tablet_manager.cc:1145] Deleting tablet ca986f81ed6848a89588366f3f400eec
0114 02:54:32.273628 (+ 64us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:32.273809 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2699855}
I20250114 02:54:32.274852 1423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/master-2/data/info.pb
I20250114 02:54:32.275269 1992 tablet_replica.cc:331] T c86562688d5b4fffb71cbdf7be0afc37 P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
W20250114 02:54:32.275315 458 catalog_manager.cc:4839] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet ca986f81ed6848a89588366f3f400eec because the tablet was not found. No further retry: Not found: Tablet not found: ca986f81ed6848a89588366f3f400eec
I20250114 02:54:32.276031 1965 ts_tablet_manager.cc:1905] T a1cf92e5275b454ca893b12f2d90d25b P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:32.277132 429 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrCUXNq/build/tsan/bin/kudu as pid 1423
I20250114 02:54:32.283200 1103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30a56fb2d941470590e17f37b90aef26" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:32.290493 1103 raft_consensus.cc:3054] T df045d80d8ce4ff194a6d2039a44bce6 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:32.281440 1205 tablet_bootstrap.cc:492] T 30a56fb2d941470590e17f37b90aef26 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:32.302599 1207 tablet_bootstrap.cc:492] T 97f4e061d3b94ee9982410b7ce135453 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:32.303020 1207 ts_tablet_manager.cc:1397] T 97f4e061d3b94ee9982410b7ce135453 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.224s user 0.006s sys 0.003s
I20250114 02:54:32.306340 1207 raft_consensus.cc:357] T 97f4e061d3b94ee9982410b7ce135453 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:32.307011 1207 raft_consensus.cc:383] T 97f4e061d3b94ee9982410b7ce135453 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:32.307343 1207 raft_consensus.cc:738] T 97f4e061d3b94ee9982410b7ce135453 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
W20250114 02:54:32.313586 751 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.66:44037, user_credentials={real_user=slave}} blocked reactor thread for 168799us
W20250114 02:54:32.317837 753 leader_election.cc:343] T 30a56fb2d941470590e17f37b90aef26 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:32.319581 2308 consensus_queue.cc:1035] T df045d80d8ce4ff194a6d2039a44bce6 P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:32.324465 1207 consensus_queue.cc:260] T 97f4e061d3b94ee9982410b7ce135453 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:32.334228 1992 raft_consensus.cc:2238] T c86562688d5b4fffb71cbdf7be0afc37 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:32.334525 1205 tablet_bootstrap.cc:654] T 30a56fb2d941470590e17f37b90aef26 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:32.334744 1992 raft_consensus.cc:2267] T c86562688d5b4fffb71cbdf7be0afc37 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:32.339799 1992 ts_tablet_manager.cc:1905] T c86562688d5b4fffb71cbdf7be0afc37 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:32.342382 1207 ts_tablet_manager.cc:1428] T 97f4e061d3b94ee9982410b7ce135453 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.039s user 0.006s sys 0.001s
I20250114 02:54:32.346000 1206 tablet_bootstrap.cc:492] T d2a7b2bee5c1440fb705b172c18a9247 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:32.346410 1206 ts_tablet_manager.cc:1397] T d2a7b2bee5c1440fb705b172c18a9247 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.744s user 0.018s sys 0.000s
I20250114 02:54:32.353700 1207 tablet_bootstrap.cc:492] T 99b940d872ce4781907517c40fe08781 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
W20250114 02:54:32.355227 457 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 961b6a25a13a4fa2a732a6383dcd8c84 (table default.Table_0943b1ca347042c9a6043006bc566701 [id=e3418787ef0a4e22979fb41d4f94ebb0], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 961b6a25a13a4fa2a732a6383dcd8c84 consensus state has no leader
I20250114 02:54:32.358260 811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "719188f639724851a5fc2b37cb60fdbb" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:32.359870 811 raft_consensus.cc:2388] T 719188f639724851a5fc2b37cb60fdbb P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 90d6c2c58bb54e0eafb365625ac33871 in current term 1: Already voted for candidate 819f9e3412e54d44abb11d8b3ca877c2 in this term.
I20250114 02:54:32.361842 959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30a56fb2d941470590e17f37b90aef26" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:32.369163 2097 consensus_queue.cc:1035] T 6e2fb05f72b743e2a02ed36b63f80b23 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:54:32.384725 959 raft_consensus.cc:3054] T fc406eab968745d893f1a8538ff61f37 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
W20250114 02:54:32.388896 898 consensus_peers.cc:487] T 6e2fb05f72b743e2a02ed36b63f80b23 P 525198593c8e47afaee54e74267b87e3 -> Peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Couldn't send request to peer 90d6c2c58bb54e0eafb365625ac33871. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250114 02:54:32.397097 1965 ts_tablet_manager.cc:1918] T a1cf92e5275b454ca893b12f2d90d25b P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20250114 02:54:32.397563 1965 log.cc:1198] T a1cf92e5275b454ca893b12f2d90d25b P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/a1cf92e5275b454ca893b12f2d90d25b
I20250114 02:54:32.398382 1965 ts_tablet_manager.cc:1939] T a1cf92e5275b454ca893b12f2d90d25b P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:32.413580 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet a1cf92e5275b454ca893b12f2d90d25b (table default.Table_b146341dc9614d3d92e92e0ecfe60405 [id=9ec503e449fb4a2a96bbb77237c2fb4f]) successfully deleted
W20250114 02:54:32.414383 456 catalog_manager.cc:4852] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet 232d6a697f744e179a1a8da492a7d227 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 232d6a697f744e179a1a8da492a7d227 already in progress: creating tablet
I20250114 02:54:32.433545 1206 raft_consensus.cc:357] T d2a7b2bee5c1440fb705b172c18a9247 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:32.434554 1206 raft_consensus.cc:383] T d2a7b2bee5c1440fb705b172c18a9247 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:32.434813 1206 raft_consensus.cc:738] T d2a7b2bee5c1440fb705b172c18a9247 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
W20250114 02:54:32.448791 1042 consensus_peers.cc:487] T 6e858a8449444ef999b9bbef612249d3 P 90d6c2c58bb54e0eafb365625ac33871 -> Peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Couldn't send request to peer 819f9e3412e54d44abb11d8b3ca877c2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6e858a8449444ef999b9bbef612249d3. This is attempt 1: this message will repeat every 5th retry.
I20250114 02:54:32.451476 1042 leader_election.cc:304] T 719188f639724851a5fc2b37cb60fdbb P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90d6c2c58bb54e0eafb365625ac33871; no voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2
I20250114 02:54:32.453054 2288 raft_consensus.cc:3054] T 719188f639724851a5fc2b37cb60fdbb P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:32.458750 2310 raft_consensus.cc:491] T 0b8989d340fa4ae992ef97e226d41110 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:32.459268 2310 raft_consensus.cc:513] T 0b8989d340fa4ae992ef97e226d41110 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:32.462431 1101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b8989d340fa4ae992ef97e226d41110" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:32.463594 2310 leader_election.cc:290] T 0b8989d340fa4ae992ef97e226d41110 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
W20250114 02:54:32.464126 753 leader_election.cc:343] T 0b8989d340fa4ae992ef97e226d41110 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:32.478613 2330 sys_catalog.cc:263] Verifying existing consensus state
I20250114 02:54:32.482707 2288 raft_consensus.cc:2743] T 719188f639724851a5fc2b37cb60fdbb P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:32.497270 2143 raft_consensus.cc:491] T fc406eab968745d893f1a8538ff61f37 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:32.497740 2143 raft_consensus.cc:513] T fc406eab968745d893f1a8538ff61f37 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:32.501601 2143 leader_election.cc:290] T fc406eab968745d893f1a8538ff61f37 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:32.505323 863 heartbeater.cc:346] Connected to a master server at 127.0.107.124:39105
I20250114 02:54:32.505508 1007 heartbeater.cc:346] Connected to a master server at 127.0.107.124:39105
I20250114 02:54:32.507742 815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc406eab968745d893f1a8538ff61f37" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:32.508462 815 raft_consensus.cc:2463] T fc406eab968745d893f1a8538ff61f37 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:32.509284 1151 heartbeater.cc:346] Connected to a master server at 127.0.107.124:39105
I20250114 02:54:32.511076 898 leader_election.cc:304] T fc406eab968745d893f1a8538ff61f37 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
I20250114 02:54:32.512042 2247 raft_consensus.cc:2798] T fc406eab968745d893f1a8538ff61f37 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:54:32.512377 2247 raft_consensus.cc:491] T fc406eab968745d893f1a8538ff61f37 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:32.512753 2247 raft_consensus.cc:3054] T fc406eab968745d893f1a8538ff61f37 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:54:32.520565 1103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc406eab968745d893f1a8538ff61f37" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:32.523630 2330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5: Bootstrap starting.
I20250114 02:54:32.583042 1990 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 330) took 1121 ms. Trace:
I20250114 02:54:32.590132 1206 consensus_queue.cc:260] T d2a7b2bee5c1440fb705b172c18a9247 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:32.610329 2330 log.cc:826] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5: Log is configured to *not* fsync() on all Append() calls
I20250114 02:54:32.664054 1990 rpcz_store.cc:276] 0114 02:54:31.461167 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:31.461294 (+ 127us) service_pool.cc:225] Handling call
0114 02:54:32.192087 (+730793us) ts_tablet_manager.cc:1145] Deleting tablet a1cf92e5275b454ca893b12f2d90d25b
0114 02:54:32.192122 (+ 35us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:32.254238 (+ 62116us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:32.264479 (+ 10241us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:32.582984 (+318505us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":85848,"tablet-delete.queue_time_us":728606}
I20250114 02:54:32.665632 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet a1cf92e5275b454ca893b12f2d90d25b (table default.Table_b146341dc9614d3d92e92e0ecfe60405 [id=9ec503e449fb4a2a96bbb77237c2fb4f]) successfully deleted
I20250114 02:54:32.666081 1990 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 332) took 1023 ms. Trace:
I20250114 02:54:32.666399 1990 rpcz_store.cc:276] 0114 02:54:31.642370 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:31.642498 (+ 128us) service_pool.cc:225] Handling call
0114 02:54:32.665716 (+1023218us) ts_tablet_manager.cc:1145] Deleting tablet 232d6a697f744e179a1a8da492a7d227
0114 02:54:32.665759 (+ 43us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:32.666028 (+ 269us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1022677}
I20250114 02:54:32.584805 2288 consensus_queue.cc:1035] T fc406eab968745d893f1a8538ff61f37 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250114 02:54:32.667475 458 catalog_manager.cc:4852] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 232d6a697f744e179a1a8da492a7d227 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 232d6a697f744e179a1a8da492a7d227 already in progress: creating tablet
W20250114 02:54:32.671695 751 leader_election.cc:343] T 30a56fb2d941470590e17f37b90aef26 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:32.672122 751 leader_election.cc:304] T 30a56fb2d941470590e17f37b90aef26 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
W20250114 02:54:32.677572 2338 log.cc:926] Time spent T df045d80d8ce4ff194a6d2039a44bce6 P 819f9e3412e54d44abb11d8b3ca877c2: Append to log took a long time: real 0.084s user 0.000s sys 0.000s
W20250114 02:54:32.684360 753 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.67:45749, user_credentials={real_user=slave}} blocked reactor thread for 51418.5us
I20250114 02:54:32.690662 1151 heartbeater.cc:346] Connected to a master server at 127.0.107.124:39105
I20250114 02:54:32.700323 2143 raft_consensus.cc:491] T 97f4e061d3b94ee9982410b7ce135453 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:32.700888 1206 ts_tablet_manager.cc:1428] T d2a7b2bee5c1440fb705b172c18a9247 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.354s user 0.004s sys 0.004s
I20250114 02:54:32.700812 2143 raft_consensus.cc:513] T 97f4e061d3b94ee9982410b7ce135453 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:32.702884 2143 leader_election.cc:290] T 97f4e061d3b94ee9982410b7ce135453 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:32.704509 813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97f4e061d3b94ee9982410b7ce135453" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:32.705106 813 raft_consensus.cc:2463] T 97f4e061d3b94ee9982410b7ce135453 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:32.706661 898 leader_election.cc:304] T 97f4e061d3b94ee9982410b7ce135453 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
I20250114 02:54:32.708531 1205 tablet_bootstrap.cc:492] T 30a56fb2d941470590e17f37b90aef26 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:32.708899 1205 ts_tablet_manager.cc:1397] T 30a56fb2d941470590e17f37b90aef26 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.428s user 0.040s sys 0.026s
I20250114 02:54:32.708910 2097 raft_consensus.cc:2798] T 97f4e061d3b94ee9982410b7ce135453 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:32.709519 2097 raft_consensus.cc:491] T 97f4e061d3b94ee9982410b7ce135453 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:32.709650 1103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97f4e061d3b94ee9982410b7ce135453" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:32.709991 2097 raft_consensus.cc:3054] T 97f4e061d3b94ee9982410b7ce135453 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:32.710425 1103 raft_consensus.cc:2463] T 97f4e061d3b94ee9982410b7ce135453 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:32.640165 2247 raft_consensus.cc:513] T fc406eab968745d893f1a8538ff61f37 P 525198593c8e47afaee54e74267b87e3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:32.713280 2247 leader_election.cc:290] T fc406eab968745d893f1a8538ff61f37 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 2 election: Requested vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:32.713945 959 raft_consensus.cc:1235] T fc406eab968745d893f1a8538ff61f37 P 525198593c8e47afaee54e74267b87e3 [term 2 FOLLOWER]: Rejecting Update request from peer 90d6c2c58bb54e0eafb365625ac33871 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20250114 02:54:32.716451 2279 consensus_queue.cc:1046] T fc406eab968745d893f1a8538ff61f37 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:32.717455 2273 raft_consensus.cc:3049] T fc406eab968745d893f1a8538ff61f37 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:54:32.717805 2273 raft_consensus.cc:738] T fc406eab968745d893f1a8538ff61f37 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Running, Role: LEADER
I20250114 02:54:32.718483 2273 consensus_queue.cc:260] T fc406eab968745d893f1a8538ff61f37 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:32.719343 2273 raft_consensus.cc:3054] T fc406eab968745d893f1a8538ff61f37 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:54:32.720899 813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc406eab968745d893f1a8538ff61f37" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:32.721418 813 raft_consensus.cc:3054] T fc406eab968745d893f1a8538ff61f37 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:54:32.728842 813 raft_consensus.cc:2463] T fc406eab968745d893f1a8538ff61f37 P 819f9e3412e54d44abb11d8b3ca877c2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 2.
I20250114 02:54:32.740435 2097 raft_consensus.cc:513] T 97f4e061d3b94ee9982410b7ce135453 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:32.742561 2097 leader_election.cc:290] T 97f4e061d3b94ee9982410b7ce135453 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Requested vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
W20250114 02:54:32.743877 751 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.66:44037, user_credentials={real_user=slave}} blocked reactor thread for 72211.7us
I20250114 02:54:32.744822 2312 raft_consensus.cc:2743] T 30a56fb2d941470590e17f37b90aef26 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:32.746178 2312 consensus_queue.cc:1035] T df045d80d8ce4ff194a6d2039a44bce6 P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:32.749449 957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b8989d340fa4ae992ef97e226d41110" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:32.753016 1103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc406eab968745d893f1a8538ff61f37" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:32.753945 1103 raft_consensus.cc:2405] T fc406eab968745d893f1a8538ff61f37 P 90d6c2c58bb54e0eafb365625ac33871 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 525198593c8e47afaee54e74267b87e3 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250114 02:54:32.755385 1206 tablet_bootstrap.cc:492] T a83bae91c29b463aa78412ce2e152ade P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:32.757203 2330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=11 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250114 02:54:32.757547 811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97f4e061d3b94ee9982410b7ce135453" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:32.758163 2330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5: Bootstrap complete.
I20250114 02:54:32.758100 811 raft_consensus.cc:3054] T 97f4e061d3b94ee9982410b7ce135453 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:32.761034 1206 tablet_bootstrap.cc:654] T a83bae91c29b463aa78412ce2e152ade P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:32.776476 1207 tablet_bootstrap.cc:654] T 99b940d872ce4781907517c40fe08781 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:32.779491 898 leader_election.cc:304] T fc406eab968745d893f1a8538ff61f37 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters: 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:32.780483 2143 raft_consensus.cc:2798] T fc406eab968745d893f1a8538ff61f37 P 525198593c8e47afaee54e74267b87e3 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 02:54:32.781548 2143 raft_consensus.cc:695] T fc406eab968745d893f1a8538ff61f37 P 525198593c8e47afaee54e74267b87e3 [term 2 LEADER]: Becoming Leader. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Running, Role: LEADER
I20250114 02:54:32.782485 2143 consensus_queue.cc:237] T fc406eab968745d893f1a8538ff61f37 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:32.797277 2330 raft_consensus.cc:357] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b303b7df086340a0b5ad6373940ae750" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 45367 } } peers { permanent_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } } peers { permanent_uuid: "87b3db09ac87436e9a60d6de5fedaff5" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } }
I20250114 02:54:32.804580 2330 raft_consensus.cc:738] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87b3db09ac87436e9a60d6de5fedaff5, State: Initialized, Role: FOLLOWER
I20250114 02:54:32.805747 2330 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b303b7df086340a0b5ad6373940ae750" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 45367 } } peers { permanent_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } } peers { permanent_uuid: "87b3db09ac87436e9a60d6de5fedaff5" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } }
W20250114 02:54:32.784602 751 leader_election.cc:343] T 0b8989d340fa4ae992ef97e226d41110 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:32.808290 751 leader_election.cc:304] T 0b8989d340fa4ae992ef97e226d41110 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:32.809017 2312 raft_consensus.cc:2743] T 0b8989d340fa4ae992ef97e226d41110 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:32.814282 2351 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b303b7df086340a0b5ad6373940ae750" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 45367 } } peers { permanent_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } } peers { permanent_uuid: "87b3db09ac87436e9a60d6de5fedaff5" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } } }
I20250114 02:54:32.815218 2351 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [sys.catalog]: This master's current role is: FOLLOWER
I20250114 02:54:32.819195 2330 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [sys.catalog]: configured and running, proceeding with master startup.
I20250114 02:54:32.828819 1205 raft_consensus.cc:357] T 30a56fb2d941470590e17f37b90aef26 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:32.829718 1205 raft_consensus.cc:383] T 30a56fb2d941470590e17f37b90aef26 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:32.830001 1205 raft_consensus.cc:738] T 30a56fb2d941470590e17f37b90aef26 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:32.830742 1205 consensus_queue.cc:260] T 30a56fb2d941470590e17f37b90aef26 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:32.834045 1103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97f4e061d3b94ee9982410b7ce135453" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:32.834631 1103 raft_consensus.cc:3054] T 97f4e061d3b94ee9982410b7ce135453 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:32.837656 1206 tablet_bootstrap.cc:492] T a83bae91c29b463aa78412ce2e152ade P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:32.838068 1206 ts_tablet_manager.cc:1397] T a83bae91c29b463aa78412ce2e152ade P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.083s user 0.015s sys 0.000s
I20250114 02:54:32.841750 1206 raft_consensus.cc:357] T a83bae91c29b463aa78412ce2e152ade P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
W20250114 02:54:32.843113 751 consensus_peers.cc:487] T 99b940d872ce4781907517c40fe08781 P 819f9e3412e54d44abb11d8b3ca877c2 -> Peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Couldn't send request to peer 525198593c8e47afaee54e74267b87e3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99b940d872ce4781907517c40fe08781. This is attempt 1: this message will repeat every 5th retry.
I20250114 02:54:32.842470 1206 raft_consensus.cc:383] T a83bae91c29b463aa78412ce2e152ade P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:32.845678 1206 raft_consensus.cc:738] T a83bae91c29b463aa78412ce2e152ade P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:32.846757 1206 consensus_queue.cc:260] T a83bae91c29b463aa78412ce2e152ade P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:32.848949 1206 ts_tablet_manager.cc:1428] T a83bae91c29b463aa78412ce2e152ade P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20250114 02:54:32.850054 1206 tablet_bootstrap.cc:492] T 5d545ac27a7f49faace0655713405cea P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:32.852147 1992 ts_tablet_manager.cc:1918] T c86562688d5b4fffb71cbdf7be0afc37 P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:32.852547 1992 log.cc:1198] T c86562688d5b4fffb71cbdf7be0afc37 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/c86562688d5b4fffb71cbdf7be0afc37
I20250114 02:54:32.853220 1992 ts_tablet_manager.cc:1939] T c86562688d5b4fffb71cbdf7be0afc37 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:32.860293 1084 tablet_service.cc:1467] Processing CreateTablet for tablet 17d76977fb874991b3dc3ae00a371130 (DEFAULT_TABLE table=default.Table_c8f09aee14204d5ea1c506f230fb81b6 [id=8d9132464cb64cc096578290ede567f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:32.861658 1084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17d76977fb874991b3dc3ae00a371130. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:32.862699 1076 tablet_service.cc:1467] Processing CreateTablet for tablet ba01a5d6a5764be78c0b817381651986 (DEFAULT_TABLE table=default.Table_d21051d46c4043488adb2d04cfb8cead [id=87438984bbfd4f89b81a4550d30130f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:32.863739 1076 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba01a5d6a5764be78c0b817381651986. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:32.863837 931 tablet_service.cc:1467] Processing CreateTablet for tablet ba01a5d6a5764be78c0b817381651986 (DEFAULT_TABLE table=default.Table_d21051d46c4043488adb2d04cfb8cead [id=87438984bbfd4f89b81a4550d30130f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:32.865003 931 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba01a5d6a5764be78c0b817381651986. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:32.864972 785 tablet_service.cc:1467] Processing CreateTablet for tablet ba01a5d6a5764be78c0b817381651986 (DEFAULT_TABLE table=default.Table_d21051d46c4043488adb2d04cfb8cead [id=87438984bbfd4f89b81a4550d30130f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:32.866103 785 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba01a5d6a5764be78c0b817381651986. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:32.867053 935 tablet_service.cc:1467] Processing CreateTablet for tablet 6a05e03fa7e1475fb42e61573c115b32 (DEFAULT_TABLE table=default.Table_1d46f092cec5487fbf47ab3d83aaeb87 [id=903117c8ff5f4ecf9913fc4fe6280465]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:32.867961 935 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a05e03fa7e1475fb42e61573c115b32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:32.869004 794 tablet_service.cc:1467] Processing CreateTablet for tablet 17d76977fb874991b3dc3ae00a371130 (DEFAULT_TABLE table=default.Table_c8f09aee14204d5ea1c506f230fb81b6 [id=8d9132464cb64cc096578290ede567f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:32.869318 1086 tablet_service.cc:1467] Processing CreateTablet for tablet 6a05e03fa7e1475fb42e61573c115b32 (DEFAULT_TABLE table=default.Table_1d46f092cec5487fbf47ab3d83aaeb87 [id=903117c8ff5f4ecf9913fc4fe6280465]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:32.869896 794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17d76977fb874991b3dc3ae00a371130. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:32.870222 1086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a05e03fa7e1475fb42e61573c115b32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:32.872944 932 tablet_service.cc:1467] Processing CreateTablet for tablet 17d76977fb874991b3dc3ae00a371130 (DEFAULT_TABLE table=default.Table_c8f09aee14204d5ea1c506f230fb81b6 [id=8d9132464cb64cc096578290ede567f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:32.873926 932 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17d76977fb874991b3dc3ae00a371130. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:32.882433 784 tablet_service.cc:1467] Processing CreateTablet for tablet 6a05e03fa7e1475fb42e61573c115b32 (DEFAULT_TABLE table=default.Table_1d46f092cec5487fbf47ab3d83aaeb87 [id=903117c8ff5f4ecf9913fc4fe6280465]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:32.883567 784 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a05e03fa7e1475fb42e61573c115b32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:32.883571 1075 tablet_service.cc:1467] Processing CreateTablet for tablet 4b83c748e50b47799b9f26b9332e1f38 (DEFAULT_TABLE table=default.Table_d0eabdfdf4214a32b3afc2c70c63eb9c [id=6ce308f2d2dd4b53bed4a3c6b973c7e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:32.884667 1075 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b83c748e50b47799b9f26b9332e1f38. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:32.886070 787 tablet_service.cc:1467] Processing CreateTablet for tablet 4b83c748e50b47799b9f26b9332e1f38 (DEFAULT_TABLE table=default.Table_d0eabdfdf4214a32b3afc2c70c63eb9c [id=6ce308f2d2dd4b53bed4a3c6b973c7e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:32.887100 787 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b83c748e50b47799b9f26b9332e1f38. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:32.887038 1081 tablet_service.cc:1467] Processing CreateTablet for tablet 18338deb62e14de49614c0c354b84814 (DEFAULT_TABLE table=default.Table_161092dfc9464fae90944e56b7ba1740 [id=c805ce6c46c948049bb54209abf69d85]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:32.887930 1081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18338deb62e14de49614c0c354b84814. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:32.888829 934 tablet_service.cc:1467] Processing CreateTablet for tablet 4b83c748e50b47799b9f26b9332e1f38 (DEFAULT_TABLE table=default.Table_d0eabdfdf4214a32b3afc2c70c63eb9c [id=6ce308f2d2dd4b53bed4a3c6b973c7e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:32.889801 934 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b83c748e50b47799b9f26b9332e1f38. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:32.903028 1207 tablet_bootstrap.cc:492] T 99b940d872ce4781907517c40fe08781 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:32.903708 1207 ts_tablet_manager.cc:1397] T 99b940d872ce4781907517c40fe08781 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.550s user 0.004s sys 0.006s
I20250114 02:54:32.906152 1207 raft_consensus.cc:357] T 99b940d872ce4781907517c40fe08781 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:32.906821 1207 raft_consensus.cc:383] T 99b940d872ce4781907517c40fe08781 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:32.907068 1207 raft_consensus.cc:738] T 99b940d872ce4781907517c40fe08781 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:32.914331 1206 tablet_bootstrap.cc:654] T 5d545ac27a7f49faace0655713405cea P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:32.932194 464 catalog_manager.cc:5526] T 38fe8829d98547fb907ef78099df3d7f P 525198593c8e47afaee54e74267b87e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 525198593c8e47afaee54e74267b87e3 (127.0.107.66). New cstate: current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:32.934352 939 tablet_service.cc:1467] Processing CreateTablet for tablet 18338deb62e14de49614c0c354b84814 (DEFAULT_TABLE table=default.Table_161092dfc9464fae90944e56b7ba1740 [id=c805ce6c46c948049bb54209abf69d85]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:32.935317 939 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18338deb62e14de49614c0c354b84814. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:32.936067 786 tablet_service.cc:1467] Processing CreateTablet for tablet 18338deb62e14de49614c0c354b84814 (DEFAULT_TABLE table=default.Table_161092dfc9464fae90944e56b7ba1740 [id=c805ce6c46c948049bb54209abf69d85]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:32.937170 786 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18338deb62e14de49614c0c354b84814. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:32.939033 464 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 38fe8829d98547fb907ef78099df3d7f (table default.Table_60450873174a4c80aa7e2122343d82be [id=9299683349354f879847b7c9d93f6072]). Expected version 1 got 0
I20250114 02:54:32.944244 464 catalog_manager.cc:5526] T edc69b66bea54625a4b15867b75986b7 P 525198593c8e47afaee54e74267b87e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 525198593c8e47afaee54e74267b87e3 (127.0.107.66). New cstate: current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:32.962373 464 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet edc69b66bea54625a4b15867b75986b7 (table default.Table_0a0bda55a4a9476c98d961822b8a19bd [id=fb2be5c1ca9444a197f3315ea9c59b09]). Expected version 1 got 0
I20250114 02:54:32.971410 811 raft_consensus.cc:2463] T 97f4e061d3b94ee9982410b7ce135453 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:32.972658 898 leader_election.cc:304] T 97f4e061d3b94ee9982410b7ce135453 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
I20250114 02:54:32.973876 1992 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 282) took 3485 ms. Trace:
I20250114 02:54:32.974181 1992 rpcz_store.cc:276] 0114 02:54:29.488822 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:29.488927 (+ 105us) service_pool.cc:225] Handling call
0114 02:54:32.275098 (+2786171us) ts_tablet_manager.cc:1145] Deleting tablet c86562688d5b4fffb71cbdf7be0afc37
0114 02:54:32.275148 (+ 50us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:32.676145 (+400997us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:32.851943 (+175798us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:32.973821 (+121878us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":58037,"tablet-delete.queue_time_us":2701188}
I20250114 02:54:32.975581 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet c86562688d5b4fffb71cbdf7be0afc37 (table default.Table_7176743e4429467481ed67254643b9f9 [id=22e634dc594d4962b404424977f2cd74]) successfully deleted
I20250114 02:54:32.992700 464 catalog_manager.cc:5526] T 7774c2087422492696a58e0075921ae7 P 525198593c8e47afaee54e74267b87e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 525198593c8e47afaee54e74267b87e3 (127.0.107.66). New cstate: current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:33.001205 464 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 719188f639724851a5fc2b37cb60fdbb (table default.Table_8f8a18d7868c44bbb4d66a0692b08186 [id=e6292728826f437cb75c524a8aef2e87]). Expected version 1 got 0
I20250114 02:54:33.006960 464 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet a52d6749ff834b62a8e20a2f45935240 (table default.Table_1c9b7896941e4289b1c12e20c210c2c1 [id=b3ee15b9f10f4d489edbde3af9034d25]). Expected version 1 got 0
I20250114 02:54:33.008527 464 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet df045d80d8ce4ff194a6d2039a44bce6 (table default.Table_2befe573a994473abe0556ba7e713686 [id=da0258578bb34ae29551fc593bf7d0b5]). Expected version 1 got 0
I20250114 02:54:33.011296 1076 tablet_service.cc:1467] Processing CreateTablet for tablet 6027bc3f9121401fbdced24aabaa2d94 (DEFAULT_TABLE table=default.Table_bbdaf710a91a429f9dd60f183ceea9f9 [id=4507c028bd074fd0a1d0231ea671b355]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:33.012501 1076 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6027bc3f9121401fbdced24aabaa2d94. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:33.029986 1103 raft_consensus.cc:2463] T 97f4e061d3b94ee9982410b7ce135453 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:33.035507 787 tablet_service.cc:1467] Processing CreateTablet for tablet 6027bc3f9121401fbdced24aabaa2d94 (DEFAULT_TABLE table=default.Table_bbdaf710a91a429f9dd60f183ceea9f9 [id=4507c028bd074fd0a1d0231ea671b355]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:33.037919 1206 tablet_bootstrap.cc:492] T 5d545ac27a7f49faace0655713405cea P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:33.038401 1206 ts_tablet_manager.cc:1397] T 5d545ac27a7f49faace0655713405cea P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.188s user 0.004s sys 0.012s
I20250114 02:54:32.907609 1207 consensus_queue.cc:260] T 99b940d872ce4781907517c40fe08781 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:33.041754 1206 raft_consensus.cc:357] T 5d545ac27a7f49faace0655713405cea P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:33.042447 1206 raft_consensus.cc:383] T 5d545ac27a7f49faace0655713405cea P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:33.042732 1206 raft_consensus.cc:738] T 5d545ac27a7f49faace0655713405cea P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:33.043509 1206 consensus_queue.cc:260] T 5d545ac27a7f49faace0655713405cea P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:33.045241 1205 ts_tablet_manager.cc:1428] T 30a56fb2d941470590e17f37b90aef26 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.336s user 0.007s sys 0.000s
I20250114 02:54:33.045540 939 tablet_service.cc:1467] Processing CreateTablet for tablet 6027bc3f9121401fbdced24aabaa2d94 (DEFAULT_TABLE table=default.Table_bbdaf710a91a429f9dd60f183ceea9f9 [id=4507c028bd074fd0a1d0231ea671b355]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:33.046906 939 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6027bc3f9121401fbdced24aabaa2d94. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:33.049633 1207 ts_tablet_manager.cc:1428] T 99b940d872ce4781907517c40fe08781 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.146s user 0.005s sys 0.000s
I20250114 02:54:33.050420 1207 tablet_bootstrap.cc:492] T 59a46bc865cc48df8e98598764fd1b54 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:33.058241 1075 tablet_service.cc:1467] Processing CreateTablet for tablet 3e89c2c12ee74060bb54f0a1dd3d1e7b (DEFAULT_TABLE table=default.Table_82cba210b70a491290d9ec065423f7c1 [id=12254cb2191f486cb23b978958c999f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:33.060678 785 tablet_service.cc:1467] Processing CreateTablet for tablet 3e89c2c12ee74060bb54f0a1dd3d1e7b (DEFAULT_TABLE table=default.Table_82cba210b70a491290d9ec065423f7c1 [id=12254cb2191f486cb23b978958c999f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:33.063062 1075 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e89c2c12ee74060bb54f0a1dd3d1e7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:33.063063 1084 tablet_service.cc:1467] Processing CreateTablet for tablet 0271201307914d1b98cb8569b3ac117e (DEFAULT_TABLE table=default.Table_36be92916bc1421bb2c3261d44b3a6db [id=d233b7100cba40aebf374397a52ed778]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:33.065433 1084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0271201307914d1b98cb8569b3ac117e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:33.065234 784 tablet_service.cc:1467] Processing CreateTablet for tablet 0271201307914d1b98cb8569b3ac117e (DEFAULT_TABLE table=default.Table_36be92916bc1421bb2c3261d44b3a6db [id=d233b7100cba40aebf374397a52ed778]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:33.068006 794 tablet_service.cc:1467] Processing CreateTablet for tablet 6d5da43a43314707bcdfde1f01efd3dc (DEFAULT_TABLE table=default.Table_403c29800daf456d93b3b60002fa6f96 [id=dcae1865f0b44cd8831a9fd0ab629415]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:33.068945 1205 tablet_bootstrap.cc:492] T 967f4d5b944543d49e87645d069b5b55 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:33.070715 934 tablet_service.cc:1467] Processing CreateTablet for tablet 3e89c2c12ee74060bb54f0a1dd3d1e7b (DEFAULT_TABLE table=default.Table_82cba210b70a491290d9ec065423f7c1 [id=12254cb2191f486cb23b978958c999f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:33.044487 2247 raft_consensus.cc:491] T 30a56fb2d941470590e17f37b90aef26 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:33.072333 935 tablet_service.cc:1467] Processing CreateTablet for tablet 0271201307914d1b98cb8569b3ac117e (DEFAULT_TABLE table=default.Table_36be92916bc1421bb2c3261d44b3a6db [id=d233b7100cba40aebf374397a52ed778]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:33.077865 468 catalog_manager.cc:5526] T d05089eae2b74f9fb23a226d358a59da P 90d6c2c58bb54e0eafb365625ac33871 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67). New cstate: current_term: 1 leader_uuid: "90d6c2c58bb54e0eafb365625ac33871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:33.082625 468 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet edc69b66bea54625a4b15867b75986b7 (table default.Table_0a0bda55a4a9476c98d961822b8a19bd [id=fb2be5c1ca9444a197f3315ea9c59b09]). Expected version 1 got 0
I20250114 02:54:33.084676 934 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e89c2c12ee74060bb54f0a1dd3d1e7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:33.080652 2247 raft_consensus.cc:513] T 30a56fb2d941470590e17f37b90aef26 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:33.086692 2247 leader_election.cc:290] T 30a56fb2d941470590e17f37b90aef26 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:33.092639 935 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0271201307914d1b98cb8569b3ac117e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:33.046329 2143 raft_consensus.cc:2798] T 97f4e061d3b94ee9982410b7ce135453 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:33.096861 2143 raft_consensus.cc:695] T 97f4e061d3b94ee9982410b7ce135453 P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Becoming Leader. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Running, Role: LEADER
I20250114 02:54:33.097775 2143 consensus_queue.cc:237] T 97f4e061d3b94ee9982410b7ce135453 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:33.107692 468 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet 38fe8829d98547fb907ef78099df3d7f (table default.Table_60450873174a4c80aa7e2122343d82be [id=9299683349354f879847b7c9d93f6072]). Expected version 1 got 0
I20250114 02:54:33.118737 2308 raft_consensus.cc:491] T d2a7b2bee5c1440fb705b172c18a9247 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:33.119210 2308 raft_consensus.cc:513] T d2a7b2bee5c1440fb705b172c18a9247 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:33.121044 2308 leader_election.cc:290] T d2a7b2bee5c1440fb705b172c18a9247 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:33.140569 1103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2a7b2bee5c1440fb705b172c18a9247" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
W20250114 02:54:33.141822 753 leader_election.cc:343] T d2a7b2bee5c1440fb705b172c18a9247 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:33.146792 465 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet edc69b66bea54625a4b15867b75986b7 (table default.Table_0a0bda55a4a9476c98d961822b8a19bd [id=fb2be5c1ca9444a197f3315ea9c59b09]). Expected version 1 got 0
I20250114 02:54:33.147375 465 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 141cede72b8741a8b0615d7c6bb823bb (table default.Table_4364d1df16d149d2ba10469ce7288d4e [id=13fc996d911b430290e37a94dd44b1ab]). Expected version 1 got 0
I20250114 02:54:33.147913 465 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 30a56fb2d941470590e17f37b90aef26 (table default.Table_0e0710e3f43e4e859ad32662704ee41c [id=66082998119e4c43a666fb43087fc622]). Expected version 1 got 0
I20250114 02:54:33.149075 2368 catalog_manager.cc:796] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250114 02:54:33.158613 1206 ts_tablet_manager.cc:1428] T 5d545ac27a7f49faace0655713405cea P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.120s user 0.004s sys 0.000s
I20250114 02:54:33.164091 794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d5da43a43314707bcdfde1f01efd3dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:33.164654 787 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6027bc3f9121401fbdced24aabaa2d94. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:33.172238 1992 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 280) took 3677 ms. Trace:
I20250114 02:54:33.172580 1992 rpcz_store.cc:276] 0114 02:54:29.487231 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:29.487347 (+ 116us) service_pool.cc:225] Handling call
0114 02:54:33.032411 (+3545064us) ts_tablet_manager.cc:1145] Deleting tablet 321b9ad6d1ac4d1c80c7a01faf2015af
0114 02:54:33.163757 (+131346us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:33.164346 (+ 589us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3458374}
W20250114 02:54:33.173604 458 catalog_manager.cc:4839] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet 321b9ad6d1ac4d1c80c7a01faf2015af because the tablet was not found. No further retry: Not found: Tablet not found: 321b9ad6d1ac4d1c80c7a01faf2015af
I20250114 02:54:33.173805 1992 tablet_replica.cc:331] T 25e85b0c5c9a4a3ca35a001359f21167 P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:33.185927 785 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e89c2c12ee74060bb54f0a1dd3d1e7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:33.188823 784 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0271201307914d1b98cb8569b3ac117e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:33.194265 2276 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" instance_seqno: 1736823246885504) as {username='slave'} at 127.0.107.67:41121; Asking this server to re-register.
I20250114 02:54:33.201937 959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2a7b2bee5c1440fb705b172c18a9247" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:33.203325 1103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30a56fb2d941470590e17f37b90aef26" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:33.205466 1207 tablet_bootstrap.cc:654] T 59a46bc865cc48df8e98598764fd1b54 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:33.193467 465 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 38fe8829d98547fb907ef78099df3d7f (table default.Table_60450873174a4c80aa7e2122343d82be [id=9299683349354f879847b7c9d93f6072]). Expected version 1 got 0
I20250114 02:54:33.212208 465 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet a52d6749ff834b62a8e20a2f45935240 (table default.Table_1c9b7896941e4289b1c12e20c210c2c1 [id=b3ee15b9f10f4d489edbde3af9034d25]). Expected version 1 got 0
I20250114 02:54:33.212424 1151 heartbeater.cc:463] Registering TS with master...
I20250114 02:54:33.213478 465 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 719188f639724851a5fc2b37cb60fdbb (table default.Table_8f8a18d7868c44bbb4d66a0692b08186 [id=e6292728826f437cb75c524a8aef2e87]). Expected version 1 got 0
W20250114 02:54:33.207044 898 leader_election.cc:343] T 30a56fb2d941470590e17f37b90aef26 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:33.223382 1084 tablet_service.cc:1467] Processing CreateTablet for tablet 6d5da43a43314707bcdfde1f01efd3dc (DEFAULT_TABLE table=default.Table_403c29800daf456d93b3b60002fa6f96 [id=dcae1865f0b44cd8831a9fd0ab629415]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:33.224419 1084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d5da43a43314707bcdfde1f01efd3dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:33.227247 1086 tablet_service.cc:1467] Processing CreateTablet for tablet fbeddf984d1f466192f60033715855ad (DEFAULT_TABLE table=default.Table_0943b1ca347042c9a6043006bc566701 [id=e3418787ef0a4e22979fb41d4f94ebb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:33.227814 932 tablet_service.cc:1467] Processing CreateTablet for tablet fbeddf984d1f466192f60033715855ad (DEFAULT_TABLE table=default.Table_0943b1ca347042c9a6043006bc566701 [id=e3418787ef0a4e22979fb41d4f94ebb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:33.227325 934 tablet_service.cc:1467] Processing CreateTablet for tablet 6d5da43a43314707bcdfde1f01efd3dc (DEFAULT_TABLE table=default.Table_403c29800daf456d93b3b60002fa6f96 [id=dcae1865f0b44cd8831a9fd0ab629415]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:33.228164 1086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbeddf984d1f466192f60033715855ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:33.231812 2281 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" instance_seqno: 1736823243139892) as {username='slave'} at 127.0.107.65:44399; Asking this server to re-register.
I20250114 02:54:33.237782 2276 ts_manager.cc:194] Registered new tserver with Master: 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:33.238943 465 catalog_manager.cc:5526] T df045d80d8ce4ff194a6d2039a44bce6 P 819f9e3412e54d44abb11d8b3ca877c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65). New cstate: current_term: 1 leader_uuid: "819f9e3412e54d44abb11d8b3ca877c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } }
W20250114 02:54:33.241138 751 leader_election.cc:343] T d2a7b2bee5c1440fb705b172c18a9247 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:33.241535 751 leader_election.cc:304] T d2a7b2bee5c1440fb705b172c18a9247 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:33.250121 1205 tablet_bootstrap.cc:654] T 967f4d5b944543d49e87645d069b5b55 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:33.248116 2308 raft_consensus.cc:2743] T d2a7b2bee5c1440fb705b172c18a9247 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:33.254393 797 tablet_service.cc:1467] Processing CreateTablet for tablet fbeddf984d1f466192f60033715855ad (DEFAULT_TABLE table=default.Table_0943b1ca347042c9a6043006bc566701 [id=e3418787ef0a4e22979fb41d4f94ebb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:33.253448 1992 raft_consensus.cc:2238] T 25e85b0c5c9a4a3ca35a001359f21167 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:33.261883 817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30a56fb2d941470590e17f37b90aef26" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:33.262406 817 raft_consensus.cc:2463] T 30a56fb2d941470590e17f37b90aef26 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:33.270920 898 leader_election.cc:304] T 30a56fb2d941470590e17f37b90aef26 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters: 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:33.280375 2097 raft_consensus.cc:2798] T 30a56fb2d941470590e17f37b90aef26 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:33.280833 2097 raft_consensus.cc:491] T 30a56fb2d941470590e17f37b90aef26 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:33.281240 2097 raft_consensus.cc:3054] T 30a56fb2d941470590e17f37b90aef26 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:33.281157 1084 tablet_service.cc:1467] Processing CreateTablet for tablet e27fe8625b104866aaca1bc9b7edf91a (DEFAULT_TABLE table=default.Table_ada937b35c344a4b8d1c0a3d52fa23e2 [id=a02e469d3f7946ff863a84d5906e0009]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:33.282127 1084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e27fe8625b104866aaca1bc9b7edf91a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:33.283753 1207 tablet_bootstrap.cc:492] T 59a46bc865cc48df8e98598764fd1b54 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:33.284358 1207 ts_tablet_manager.cc:1397] T 59a46bc865cc48df8e98598764fd1b54 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.234s user 0.009s sys 0.001s
I20250114 02:54:33.286433 1207 raft_consensus.cc:357] T 59a46bc865cc48df8e98598764fd1b54 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:33.287056 1207 raft_consensus.cc:383] T 59a46bc865cc48df8e98598764fd1b54 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:33.287295 1207 raft_consensus.cc:738] T 59a46bc865cc48df8e98598764fd1b54 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:33.289217 932 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbeddf984d1f466192f60033715855ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:33.296829 934 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d5da43a43314707bcdfde1f01efd3dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:33.305680 465 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet df045d80d8ce4ff194a6d2039a44bce6 (table default.Table_2befe573a994473abe0556ba7e713686 [id=da0258578bb34ae29551fc593bf7d0b5]). Expected version 1 got 0
I20250114 02:54:33.307632 2351 raft_consensus.cc:491] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:33.307667 2281 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "525198593c8e47afaee54e74267b87e3" instance_seqno: 1736823245009848) as {username='slave'} at 127.0.107.66:60145; Asking this server to re-register.
I20250114 02:54:33.308197 2351 raft_consensus.cc:513] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b303b7df086340a0b5ad6373940ae750" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 45367 } } peers { permanent_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } } peers { permanent_uuid: "87b3db09ac87436e9a60d6de5fedaff5" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } }
I20250114 02:54:33.301106 2097 raft_consensus.cc:513] T 30a56fb2d941470590e17f37b90aef26 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:33.311640 481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "87b3db09ac87436e9a60d6de5fedaff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "b303b7df086340a0b5ad6373940ae750" is_pre_election: true
I20250114 02:54:33.313237 2097 leader_election.cc:290] T 30a56fb2d941470590e17f37b90aef26 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Requested vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:33.313694 2097 raft_consensus.cc:491] T 719188f639724851a5fc2b37cb60fdbb P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:33.313753 2351 leader_election.cc:290] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b303b7df086340a0b5ad6373940ae750 (127.0.107.126:45367), 687d4ff6d3cb42cf90a965070b0d8a70 (127.0.107.125:46837)
I20250114 02:54:33.314168 2097 raft_consensus.cc:513] T 719188f639724851a5fc2b37cb60fdbb P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:33.316051 2097 leader_election.cc:290] T 719188f639724851a5fc2b37cb60fdbb P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:33.302461 939 tablet_service.cc:1467] Processing CreateTablet for tablet e27fe8625b104866aaca1bc9b7edf91a (DEFAULT_TABLE table=default.Table_ada937b35c344a4b8d1c0a3d52fa23e2 [id=a02e469d3f7946ff863a84d5906e0009]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:33.323336 939 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e27fe8625b104866aaca1bc9b7edf91a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:33.331636 932 tablet_service.cc:1467] Processing CreateTablet for tablet dc1e0e79ccec4c79a10d02af9d2062a3 (DEFAULT_TABLE table=default.Table_a573620245b049398a53d6cb8799ca39 [id=81e2d41193404732b5db92e1357b8c81]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:33.332809 932 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc1e0e79ccec4c79a10d02af9d2062a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:33.332949 1103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30a56fb2d941470590e17f37b90aef26" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:33.336349 1103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "719188f639724851a5fc2b37cb60fdbb" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:33.336941 1103 raft_consensus.cc:2463] T 719188f639724851a5fc2b37cb60fdbb P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:33.337837 863 heartbeater.cc:463] Registering TS with master...
W20250114 02:54:33.339956 898 leader_election.cc:343] T 30a56fb2d941470590e17f37b90aef26 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:33.341557 898 leader_election.cc:304] T 719188f639724851a5fc2b37cb60fdbb P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:33.342249 2143 raft_consensus.cc:2798] T 719188f639724851a5fc2b37cb60fdbb P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:54:33.342629 2143 raft_consensus.cc:491] T 719188f639724851a5fc2b37cb60fdbb P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:33.343014 2143 raft_consensus.cc:3054] T 719188f639724851a5fc2b37cb60fdbb P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:54:33.349099 797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbeddf984d1f466192f60033715855ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:33.349186 813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30a56fb2d941470590e17f37b90aef26" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:33.349761 813 raft_consensus.cc:3054] T 30a56fb2d941470590e17f37b90aef26 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:33.352677 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "719188f639724851a5fc2b37cb60fdbb" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:33.357420 818 raft_consensus.cc:2463] T 719188f639724851a5fc2b37cb60fdbb P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:33.357571 1992 raft_consensus.cc:2267] T 25e85b0c5c9a4a3ca35a001359f21167 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:33.359090 1206 tablet_bootstrap.cc:492] T 612987f3e4ea4a02932ceefa0d740425 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:33.363559 1103 raft_consensus.cc:3054] T 99b940d872ce4781907517c40fe08781 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:33.372306 1084 tablet_service.cc:1467] Processing CreateTablet for tablet dc1e0e79ccec4c79a10d02af9d2062a3 (DEFAULT_TABLE table=default.Table_a573620245b049398a53d6cb8799ca39 [id=81e2d41193404732b5db92e1357b8c81]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:33.373320 1084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc1e0e79ccec4c79a10d02af9d2062a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:33.374486 1205 tablet_bootstrap.cc:492] T 967f4d5b944543d49e87645d069b5b55 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:33.374877 1205 ts_tablet_manager.cc:1397] T 967f4d5b944543d49e87645d069b5b55 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.306s user 0.005s sys 0.007s
I20250114 02:54:33.377715 1205 raft_consensus.cc:357] T 967f4d5b944543d49e87645d069b5b55 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:33.378348 1205 raft_consensus.cc:383] T 967f4d5b944543d49e87645d069b5b55 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:33.378603 1205 raft_consensus.cc:738] T 967f4d5b944543d49e87645d069b5b55 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:33.390779 1007 heartbeater.cc:463] Registering TS with master...
I20250114 02:54:33.392498 1207 consensus_queue.cc:260] T 59a46bc865cc48df8e98598764fd1b54 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:33.394868 1207 ts_tablet_manager.cc:1428] T 59a46bc865cc48df8e98598764fd1b54 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.110s user 0.004s sys 0.001s
I20250114 02:54:33.403080 2281 ts_manager.cc:194] Registered new tserver with Master: 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:33.412521 469 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 38fe8829d98547fb907ef78099df3d7f (table default.Table_60450873174a4c80aa7e2122343d82be [id=9299683349354f879847b7c9d93f6072]). Expected version 1 got 0
I20250114 02:54:33.436038 2143 raft_consensus.cc:513] T 719188f639724851a5fc2b37cb60fdbb P 525198593c8e47afaee54e74267b87e3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:33.441360 1207 tablet_bootstrap.cc:492] T ed7f575653c848d7b076c17c38c4a244 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:33.452193 1102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "719188f639724851a5fc2b37cb60fdbb" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:33.452802 1102 raft_consensus.cc:3054] T 719188f639724851a5fc2b37cb60fdbb P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:54:33.455550 554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "87b3db09ac87436e9a60d6de5fedaff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" is_pre_election: true
I20250114 02:54:33.457803 2204 leader_election.cc:304] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87b3db09ac87436e9a60d6de5fedaff5; no voters: 687d4ff6d3cb42cf90a965070b0d8a70, b303b7df086340a0b5ad6373940ae750
I20250114 02:54:33.459062 2351 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:54:33.460491 1992 ts_tablet_manager.cc:1905] T 25e85b0c5c9a4a3ca35a001359f21167 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:33.461337 1206 tablet_bootstrap.cc:654] T 612987f3e4ea4a02932ceefa0d740425 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:33.461159 785 tablet_service.cc:1467] Processing CreateTablet for tablet e27fe8625b104866aaca1bc9b7edf91a (DEFAULT_TABLE table=default.Table_ada937b35c344a4b8d1c0a3d52fa23e2 [id=a02e469d3f7946ff863a84d5906e0009]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:33.462219 785 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e27fe8625b104866aaca1bc9b7edf91a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:33.464542 1102 raft_consensus.cc:2463] T 719188f639724851a5fc2b37cb60fdbb P 90d6c2c58bb54e0eafb365625ac33871 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 2.
W20250114 02:54:33.464624 458 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 6027bc3f9121401fbdced24aabaa2d94 (table default.Table_bbdaf710a91a429f9dd60f183ceea9f9 [id=4507c028bd074fd0a1d0231ea671b355], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 6027bc3f9121401fbdced24aabaa2d94 consensus state has no leader
I20250114 02:54:33.465701 786 tablet_service.cc:1467] Processing CreateTablet for tablet dc1e0e79ccec4c79a10d02af9d2062a3 (DEFAULT_TABLE table=default.Table_a573620245b049398a53d6cb8799ca39 [id=81e2d41193404732b5db92e1357b8c81]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:33.466666 786 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc1e0e79ccec4c79a10d02af9d2062a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:33.468703 898 leader_election.cc:304] T 719188f639724851a5fc2b37cb60fdbb P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:33.469449 2097 raft_consensus.cc:2798] T 719188f639724851a5fc2b37cb60fdbb P 525198593c8e47afaee54e74267b87e3 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 02:54:33.469882 2097 raft_consensus.cc:695] T 719188f639724851a5fc2b37cb60fdbb P 525198593c8e47afaee54e74267b87e3 [term 2 LEADER]: Becoming Leader. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Running, Role: LEADER
I20250114 02:54:33.470582 2097 consensus_queue.cc:237] T 719188f639724851a5fc2b37cb60fdbb P 525198593c8e47afaee54e74267b87e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:33.471545 813 raft_consensus.cc:2463] T 30a56fb2d941470590e17f37b90aef26 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:33.476703 811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "719188f639724851a5fc2b37cb60fdbb" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:33.477257 811 raft_consensus.cc:3054] T 719188f639724851a5fc2b37cb60fdbb P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:54:33.478559 1992 ts_tablet_manager.cc:1918] T 25e85b0c5c9a4a3ca35a001359f21167 P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:33.478950 1992 log.cc:1198] T 25e85b0c5c9a4a3ca35a001359f21167 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/25e85b0c5c9a4a3ca35a001359f21167
I20250114 02:54:33.479653 1992 ts_tablet_manager.cc:1939] T 25e85b0c5c9a4a3ca35a001359f21167 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:33.449198 469 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet edc69b66bea54625a4b15867b75986b7 (table default.Table_0a0bda55a4a9476c98d961822b8a19bd [id=fb2be5c1ca9444a197f3315ea9c59b09]). Expected version 1 got 0
I20250114 02:54:33.482841 469 catalog_manager.cc:5526] T a52d6749ff834b62a8e20a2f45935240 P 525198593c8e47afaee54e74267b87e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 525198593c8e47afaee54e74267b87e3 (127.0.107.66). New cstate: current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:33.469601 1205 consensus_queue.cc:260] T 967f4d5b944543d49e87645d069b5b55 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:33.450507 2143 leader_election.cc:290] T 719188f639724851a5fc2b37cb60fdbb P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 2 election: Requested vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:33.489670 1205 ts_tablet_manager.cc:1428] T 967f4d5b944543d49e87645d069b5b55 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.115s user 0.005s sys 0.000s
I20250114 02:54:33.490574 1205 tablet_bootstrap.cc:492] T 141cede72b8741a8b0615d7c6bb823bb P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:33.496035 898 leader_election.cc:304] T 30a56fb2d941470590e17f37b90aef26 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters: 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:33.496809 2097 raft_consensus.cc:2798] T 30a56fb2d941470590e17f37b90aef26 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:33.497249 2097 raft_consensus.cc:695] T 30a56fb2d941470590e17f37b90aef26 P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Becoming Leader. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Running, Role: LEADER
I20250114 02:54:33.498080 2097 consensus_queue.cc:237] T 30a56fb2d941470590e17f37b90aef26 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:33.508735 2383 tablet.cc:1721] T 99b940d872ce4781907517c40fe08781 P 90d6c2c58bb54e0eafb365625ac33871: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250114 02:54:33.484062 469 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet a52d6749ff834b62a8e20a2f45935240 (table default.Table_1c9b7896941e4289b1c12e20c210c2c1 [id=b3ee15b9f10f4d489edbde3af9034d25]). Expected version 1 got 0
I20250114 02:54:33.512200 469 catalog_manager.cc:5526] T 6e2fb05f72b743e2a02ed36b63f80b23 P 525198593c8e47afaee54e74267b87e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 525198593c8e47afaee54e74267b87e3 (127.0.107.66). New cstate: current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:33.513140 2375 raft_consensus.cc:491] T 5d545ac27a7f49faace0655713405cea P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:33.513564 2375 raft_consensus.cc:513] T 5d545ac27a7f49faace0655713405cea P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:33.514812 1206 tablet_bootstrap.cc:492] T 612987f3e4ea4a02932ceefa0d740425 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:33.515236 1206 ts_tablet_manager.cc:1397] T 612987f3e4ea4a02932ceefa0d740425 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.157s user 0.014s sys 0.000s
I20250114 02:54:33.515373 2375 leader_election.cc:290] T 5d545ac27a7f49faace0655713405cea P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:33.517488 1102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d545ac27a7f49faace0655713405cea" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:33.518617 959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d545ac27a7f49faace0655713405cea" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
W20250114 02:54:33.520548 751 leader_election.cc:343] T 5d545ac27a7f49faace0655713405cea P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
W20250114 02:54:33.521508 753 leader_election.cc:343] T 5d545ac27a7f49faace0655713405cea P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:33.521871 753 leader_election.cc:304] T 5d545ac27a7f49faace0655713405cea P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:33.522523 2375 raft_consensus.cc:2743] T 5d545ac27a7f49faace0655713405cea P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:33.523751 469 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 719188f639724851a5fc2b37cb60fdbb (table default.Table_8f8a18d7868c44bbb4d66a0692b08186 [id=e6292728826f437cb75c524a8aef2e87]). Expected version 1 got 0
I20250114 02:54:33.525338 811 raft_consensus.cc:2463] T 719188f639724851a5fc2b37cb60fdbb P 819f9e3412e54d44abb11d8b3ca877c2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 2.
I20250114 02:54:33.537182 469 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet df045d80d8ce4ff194a6d2039a44bce6 (table default.Table_2befe573a994473abe0556ba7e713686 [id=da0258578bb34ae29551fc593bf7d0b5]). Expected version 1 got 0
I20250114 02:54:33.539618 1206 raft_consensus.cc:357] T 612987f3e4ea4a02932ceefa0d740425 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:33.540664 2281 ts_manager.cc:194] Registered new tserver with Master: 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:33.540707 1206 raft_consensus.cc:383] T 612987f3e4ea4a02932ceefa0d740425 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:33.540949 1206 raft_consensus.cc:738] T 612987f3e4ea4a02932ceefa0d740425 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:33.541528 1206 consensus_queue.cc:260] T 612987f3e4ea4a02932ceefa0d740425 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:33.546170 1207 tablet_bootstrap.cc:654] T ed7f575653c848d7b076c17c38c4a244 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
W20250114 02:54:33.552691 462 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fbeddf984d1f466192f60033715855ad consensus state has no leader
I20250114 02:54:33.553145 462 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:33540 (request call id 296) took 3867 ms. Trace:
I20250114 02:54:33.553221 1206 ts_tablet_manager.cc:1428] T 612987f3e4ea4a02932ceefa0d740425 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.038s user 0.001s sys 0.004s
I20250114 02:54:33.554894 811 raft_consensus.cc:1270] T fc406eab968745d893f1a8538ff61f37 P 819f9e3412e54d44abb11d8b3ca877c2 [term 2 FOLLOWER]: Refusing update from remote peer 525198593c8e47afaee54e74267b87e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250114 02:54:33.553404 462 rpcz_store.cc:276] 0114 02:54:29.685521 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:29.824009 (+138488us) service_pool.cc:225] Handling call
0114 02:54:29.825039 (+ 1030us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0114 02:54:29.826199 (+ 1160us) catalog_manager.cc:3627] Apply alter schema
0114 02:54:29.826767 (+ 568us) catalog_manager.cc:3659] Acquired catalog manager lock
0114 02:54:29.826952 (+ 185us) catalog_manager.cc:3817] Updating metadata on disk
0114 02:54:29.934577 (+107625us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:29.934607 (+ 30us) write_op.cc:454] Released schema lock
0114 02:54:29.934825 (+ 218us) catalog_manager.cc:3848] Committing alterations to in-memory state
0114 02:54:33.553092 (+3618267us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:29.860150 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:29.860635 (+ 485us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:29.860667 (+ 32us) write_op.cc:435] Acquired schema lock
0114 02:54:29.860689 (+ 22us) tablet.cc:659] Decoding operations
0114 02:54:29.861000 (+ 311us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:29.861041 (+ 41us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:29.861065 (+ 24us) tablet.cc:682] Acquiring locks for 1 operations
0114 02:54:29.861256 (+ 191us) tablet.cc:698] Row locks acquired
0114 02:54:29.861277 (+ 21us) write_op.cc:260] PREPARE: finished
0114 02:54:29.861407 (+ 130us) write_op.cc:270] Start()
0114 02:54:29.861513 (+ 106us) write_op.cc:276] Timestamp: P: 1736823269861331 usec, L: 0
0114 02:54:29.861533 (+ 20us) op_driver.cc:348] REPLICATION: starting
0114 02:54:29.862100 (+ 567us) log.cc:844] Serialized 405 byte log entry
0114 02:54:29.871257 (+ 9157us) op_driver.cc:464] REPLICATION: finished
0114 02:54:29.871653 (+ 396us) write_op.cc:301] APPLY: starting
0114 02:54:29.871714 (+ 61us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:29.871817 (+ 103us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:29.871924 (+ 107us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:29.872094 (+ 170us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:29.872250 (+ 156us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:29.872301 (+ 51us) write_op.cc:312] APPLY: finished
0114 02:54:29.872507 (+ 206us) log.cc:844] Serialized 23 byte log entry
0114 02:54:29.872653 (+ 146us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:29.872748 (+ 95us) write_op.cc:454] Released schema lock
0114 02:54:29.872781 (+ 33us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":248,"apply.run_cpu_time_us":1299,"apply.run_wall_time_us":1299,"num_ops":1,"prepare.queue_time_us":32113,"prepare.run_cpu_time_us":2271,"prepare.run_wall_time_us":18285,"raft.queue_time_us":81,"raft.run_cpu_time_us":563,"raft.run_wall_time_us":564,"replication_time_us":9635}]]}
I20250114 02:54:33.571355 1205 tablet_bootstrap.cc:654] T 141cede72b8741a8b0615d7c6bb823bb P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:33.573444 1206 tablet_bootstrap.cc:492] T 0b8361cd538046e0bcb4aac821748f67 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:33.576015 1992 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 284) took 3927 ms. Trace:
I20250114 02:54:33.576382 1992 rpcz_store.cc:276] 0114 02:54:29.649122 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:29.684048 (+ 34926us) service_pool.cc:225] Handling call
0114 02:54:33.173654 (+3489606us) ts_tablet_manager.cc:1145] Deleting tablet 25e85b0c5c9a4a3ca35a001359f21167
0114 02:54:33.173707 (+ 53us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:33.471702 (+297995us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:33.478361 (+ 6659us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:33.575878 (+ 97517us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":28891,"tablet-delete.queue_time_us":3488942}
I20250114 02:54:33.577790 1992 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 289) took 3380 ms. Trace:
I20250114 02:54:33.577898 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet 25e85b0c5c9a4a3ca35a001359f21167 (table default.Table_bcf55148b6bc413e8d4e8e31996ff32b [id=d6c16b93fcf5427f9876082d1996a31f]) successfully deleted
I20250114 02:54:33.578107 1992 rpcz_store.cc:276] 0114 02:54:30.196921 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:30.197044 (+ 123us) service_pool.cc:225] Handling call
0114 02:54:33.577399 (+3380355us) ts_tablet_manager.cc:1145] Deleting tablet 4ab79e0a576843e89a2d524608c5b7f4
0114 02:54:33.577465 (+ 66us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:33.577744 (+ 279us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3379571}
I20250114 02:54:33.579051 1992 tablet_replica.cc:331] T 3161f75bc4e84000af022faa9804c810 P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
W20250114 02:54:33.579226 458 catalog_manager.cc:4852] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet 4ab79e0a576843e89a2d524608c5b7f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4ab79e0a576843e89a2d524608c5b7f4 already in progress: creating tablet
I20250114 02:54:33.579833 1992 raft_consensus.cc:2238] T 3161f75bc4e84000af022faa9804c810 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:33.590834 2143 consensus_queue.cc:1035] T fc406eab968745d893f1a8538ff61f37 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:33.596473 1206 tablet_bootstrap.cc:654] T 0b8361cd538046e0bcb4aac821748f67 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:33.633350 1992 raft_consensus.cc:2267] T 3161f75bc4e84000af022faa9804c810 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:33.633440 2374 raft_consensus.cc:491] T a83bae91c29b463aa78412ce2e152ade P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:33.633879 2374 raft_consensus.cc:513] T a83bae91c29b463aa78412ce2e152ade P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:33.635401 1992 ts_tablet_manager.cc:1905] T 3161f75bc4e84000af022faa9804c810 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:33.635774 2374 leader_election.cc:290] T a83bae91c29b463aa78412ce2e152ade P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:33.638088 1103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83bae91c29b463aa78412ce2e152ade" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
W20250114 02:54:33.639305 753 leader_election.cc:343] T a83bae91c29b463aa78412ce2e152ade P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:33.646112 959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83bae91c29b463aa78412ce2e152ade" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
W20250114 02:54:33.650719 751 leader_election.cc:343] T a83bae91c29b463aa78412ce2e152ade P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:33.651126 751 leader_election.cc:304] T a83bae91c29b463aa78412ce2e152ade P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:33.652068 2374 raft_consensus.cc:2743] T a83bae91c29b463aa78412ce2e152ade P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:33.657346 1207 tablet_bootstrap.cc:492] T ed7f575653c848d7b076c17c38c4a244 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:33.657716 1207 ts_tablet_manager.cc:1397] T ed7f575653c848d7b076c17c38c4a244 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.217s user 0.003s sys 0.005s
I20250114 02:54:33.659989 1207 raft_consensus.cc:357] T ed7f575653c848d7b076c17c38c4a244 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:33.660800 1207 raft_consensus.cc:383] T ed7f575653c848d7b076c17c38c4a244 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:33.661049 1207 raft_consensus.cc:738] T ed7f575653c848d7b076c17c38c4a244 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:33.661648 1207 consensus_queue.cc:260] T ed7f575653c848d7b076c17c38c4a244 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:33.666031 1207 ts_tablet_manager.cc:1428] T ed7f575653c848d7b076c17c38c4a244 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20250114 02:54:33.668610 1206 tablet_bootstrap.cc:492] T 0b8361cd538046e0bcb4aac821748f67 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:33.669018 1206 ts_tablet_manager.cc:1397] T 0b8361cd538046e0bcb4aac821748f67 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.096s user 0.005s sys 0.004s
I20250114 02:54:33.671420 471 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet df045d80d8ce4ff194a6d2039a44bce6 (table default.Table_2befe573a994473abe0556ba7e713686 [id=da0258578bb34ae29551fc593bf7d0b5]). Expected version 1 got 0
I20250114 02:54:33.673156 471 catalog_manager.cc:5526] T fc406eab968745d893f1a8538ff61f37 P 90d6c2c58bb54e0eafb365625ac33871 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67). New cstate: current_term: 1 leader_uuid: "90d6c2c58bb54e0eafb365625ac33871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:33.683679 471 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet a52d6749ff834b62a8e20a2f45935240 (table default.Table_1c9b7896941e4289b1c12e20c210c2c1 [id=b3ee15b9f10f4d489edbde3af9034d25]). Expected version 1 got 0
I20250114 02:54:33.671164 1206 raft_consensus.cc:357] T 0b8361cd538046e0bcb4aac821748f67 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:33.686411 1206 raft_consensus.cc:383] T 0b8361cd538046e0bcb4aac821748f67 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:33.686757 1206 raft_consensus.cc:738] T 0b8361cd538046e0bcb4aac821748f67 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:33.688424 2097 raft_consensus.cc:491] T 967f4d5b944543d49e87645d069b5b55 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:33.688827 2097 raft_consensus.cc:513] T 967f4d5b944543d49e87645d069b5b55 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:33.690726 2097 leader_election.cc:290] T 967f4d5b944543d49e87645d069b5b55 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:33.694396 811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "967f4d5b944543d49e87645d069b5b55" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:33.695070 811 raft_consensus.cc:2463] T 967f4d5b944543d49e87645d069b5b55 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:33.697081 898 leader_election.cc:304] T 967f4d5b944543d49e87645d069b5b55 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
I20250114 02:54:33.698021 2247 raft_consensus.cc:2798] T 967f4d5b944543d49e87645d069b5b55 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:33.698341 2247 raft_consensus.cc:491] T 967f4d5b944543d49e87645d069b5b55 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:33.698638 2247 raft_consensus.cc:3054] T 967f4d5b944543d49e87645d069b5b55 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:33.701026 471 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet edc69b66bea54625a4b15867b75986b7 (table default.Table_0a0bda55a4a9476c98d961822b8a19bd [id=fb2be5c1ca9444a197f3315ea9c59b09]). Expected version 1 got 0
I20250114 02:54:33.698854 1206 consensus_queue.cc:260] T 0b8361cd538046e0bcb4aac821748f67 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:33.703871 1206 ts_tablet_manager.cc:1428] T 0b8361cd538046e0bcb4aac821748f67 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.035s user 0.005s sys 0.000s
I20250114 02:54:33.705704 2247 raft_consensus.cc:513] T 967f4d5b944543d49e87645d069b5b55 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:33.707391 2247 leader_election.cc:290] T 967f4d5b944543d49e87645d069b5b55 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Requested vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:33.708866 471 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet 38fe8829d98547fb907ef78099df3d7f (table default.Table_60450873174a4c80aa7e2122343d82be [id=9299683349354f879847b7c9d93f6072]). Expected version 1 got 0
I20250114 02:54:33.709851 1205 tablet_bootstrap.cc:492] T 141cede72b8741a8b0615d7c6bb823bb P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:33.710330 1205 ts_tablet_manager.cc:1397] T 141cede72b8741a8b0615d7c6bb823bb P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.220s user 0.009s sys 0.000s
I20250114 02:54:33.713011 1205 raft_consensus.cc:357] T 141cede72b8741a8b0615d7c6bb823bb P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:33.714740 1206 tablet_bootstrap.cc:492] T 670dc7ff15d644e18c296eeb2a6b99e1 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:33.723810 1206 tablet_bootstrap.cc:654] T 670dc7ff15d644e18c296eeb2a6b99e1 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:33.737397 811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "967f4d5b944543d49e87645d069b5b55" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:33.738072 811 raft_consensus.cc:3054] T 967f4d5b944543d49e87645d069b5b55 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:33.745806 811 raft_consensus.cc:2463] T 967f4d5b944543d49e87645d069b5b55 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:33.755503 1206 tablet_bootstrap.cc:492] T 670dc7ff15d644e18c296eeb2a6b99e1 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:33.755544 898 leader_election.cc:304] T 967f4d5b944543d49e87645d069b5b55 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
I20250114 02:54:33.756125 1206 ts_tablet_manager.cc:1397] T 670dc7ff15d644e18c296eeb2a6b99e1 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.042s user 0.006s sys 0.007s
I20250114 02:54:33.757735 2143 raft_consensus.cc:2798] T 967f4d5b944543d49e87645d069b5b55 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:33.758140 2143 raft_consensus.cc:695] T 967f4d5b944543d49e87645d069b5b55 P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Becoming Leader. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Running, Role: LEADER
I20250114 02:54:33.758888 2143 consensus_queue.cc:237] T 967f4d5b944543d49e87645d069b5b55 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:33.759326 1206 raft_consensus.cc:357] T 670dc7ff15d644e18c296eeb2a6b99e1 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:33.760164 1206 raft_consensus.cc:383] T 670dc7ff15d644e18c296eeb2a6b99e1 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:33.760470 1206 raft_consensus.cc:738] T 670dc7ff15d644e18c296eeb2a6b99e1 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:33.761109 1206 consensus_queue.cc:260] T 670dc7ff15d644e18c296eeb2a6b99e1 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:33.769647 1206 ts_tablet_manager.cc:1428] T 670dc7ff15d644e18c296eeb2a6b99e1 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.013s user 0.001s sys 0.003s
I20250114 02:54:33.777244 1206 tablet_bootstrap.cc:492] T be7aad41ccdd4ef9bcfe8a985155708f P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:33.785233 1206 tablet_bootstrap.cc:654] T be7aad41ccdd4ef9bcfe8a985155708f P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:33.796898 1106 raft_consensus.cc:1270] T fc406eab968745d893f1a8538ff61f37 P 90d6c2c58bb54e0eafb365625ac33871 [term 2 FOLLOWER]: Refusing update from remote peer 525198593c8e47afaee54e74267b87e3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20250114 02:54:33.797204 1106 pending_rounds.cc:77] T fc406eab968745d893f1a8538ff61f37 P 90d6c2c58bb54e0eafb365625ac33871: Aborting all ops after (but not including) 0
I20250114 02:54:33.797448 1106 pending_rounds.cc:99] T fc406eab968745d893f1a8538ff61f37 P 90d6c2c58bb54e0eafb365625ac33871: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20250114 02:54:33.797344 1102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "967f4d5b944543d49e87645d069b5b55" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:33.797663 1106 raft_consensus.cc:2883] T fc406eab968745d893f1a8538ff61f37 P 90d6c2c58bb54e0eafb365625ac33871 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250114 02:54:33.713640 1205 raft_consensus.cc:383] T 141cede72b8741a8b0615d7c6bb823bb P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:33.800443 1205 raft_consensus.cc:738] T 141cede72b8741a8b0615d7c6bb823bb P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:33.801069 1205 consensus_queue.cc:260] T 141cede72b8741a8b0615d7c6bb823bb P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
W20250114 02:54:33.802660 898 leader_election.cc:343] T 967f4d5b944543d49e87645d069b5b55 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:33.804376 2247 consensus_queue.cc:1035] T fc406eab968745d893f1a8538ff61f37 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:33.808499 2389 raft_consensus.cc:491] T 59a46bc865cc48df8e98598764fd1b54 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:33.808538 1099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "967f4d5b944543d49e87645d069b5b55" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:33.808944 2389 raft_consensus.cc:513] T 59a46bc865cc48df8e98598764fd1b54 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:33.810801 2389 leader_election.cc:290] T 59a46bc865cc48df8e98598764fd1b54 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:33.812703 957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59a46bc865cc48df8e98598764fd1b54" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:33.813251 957 raft_consensus.cc:2463] T 59a46bc865cc48df8e98598764fd1b54 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 0.
I20250114 02:54:33.815368 1040 leader_election.cc:304] T 59a46bc865cc48df8e98598764fd1b54 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:33.816291 2279 raft_consensus.cc:2798] T 59a46bc865cc48df8e98598764fd1b54 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:33.816623 2279 raft_consensus.cc:491] T 59a46bc865cc48df8e98598764fd1b54 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:33.817026 2279 raft_consensus.cc:3054] T 59a46bc865cc48df8e98598764fd1b54 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
W20250114 02:54:33.824932 1042 leader_election.cc:343] T 59a46bc865cc48df8e98598764fd1b54 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Not found: Tablet not found: 59a46bc865cc48df8e98598764fd1b54
I20250114 02:54:33.825155 2279 raft_consensus.cc:513] T 59a46bc865cc48df8e98598764fd1b54 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
W20250114 02:54:33.828300 1042 leader_election.cc:343] T 59a46bc865cc48df8e98598764fd1b54 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Not found: Tablet not found: 59a46bc865cc48df8e98598764fd1b54
W20250114 02:54:33.831009 898 leader_election.cc:343] T 967f4d5b944543d49e87645d069b5b55 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:33.841024 1207 tablet_bootstrap.cc:492] T 3161f75bc4e84000af022faa9804c810 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:33.841737 2279 leader_election.cc:290] T 59a46bc865cc48df8e98598764fd1b54 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Requested vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:33.843127 957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59a46bc865cc48df8e98598764fd1b54" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
I20250114 02:54:33.843657 957 raft_consensus.cc:3054] T 59a46bc865cc48df8e98598764fd1b54 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:33.849685 1205 ts_tablet_manager.cc:1428] T 141cede72b8741a8b0615d7c6bb823bb P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.139s user 0.006s sys 0.000s
I20250114 02:54:33.854130 957 raft_consensus.cc:2463] T 59a46bc865cc48df8e98598764fd1b54 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1.
I20250114 02:54:33.868844 1205 tablet_bootstrap.cc:492] T f40c66314342453b898da3ac0e7e86bf P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:33.869844 1040 leader_election.cc:304] T 59a46bc865cc48df8e98598764fd1b54 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters: 819f9e3412e54d44abb11d8b3ca877c2
I20250114 02:54:33.870831 2279 raft_consensus.cc:2798] T 59a46bc865cc48df8e98598764fd1b54 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:33.871272 2279 raft_consensus.cc:695] T 59a46bc865cc48df8e98598764fd1b54 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Becoming Leader. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Running, Role: LEADER
I20250114 02:54:33.872042 2279 consensus_queue.cc:237] T 59a46bc865cc48df8e98598764fd1b54 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:33.882284 1206 tablet_bootstrap.cc:492] T be7aad41ccdd4ef9bcfe8a985155708f P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:33.882730 1206 ts_tablet_manager.cc:1397] T be7aad41ccdd4ef9bcfe8a985155708f P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.106s user 0.009s sys 0.004s
I20250114 02:54:33.890297 1206 raft_consensus.cc:357] T be7aad41ccdd4ef9bcfe8a985155708f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:33.891006 1206 raft_consensus.cc:383] T be7aad41ccdd4ef9bcfe8a985155708f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:33.891273 1206 raft_consensus.cc:738] T be7aad41ccdd4ef9bcfe8a985155708f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:33.881402 1207 tablet_bootstrap.cc:654] T 3161f75bc4e84000af022faa9804c810 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:33.905622 1992 ts_tablet_manager.cc:1918] T 3161f75bc4e84000af022faa9804c810 P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:33.905990 1992 log.cc:1198] T 3161f75bc4e84000af022faa9804c810 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/3161f75bc4e84000af022faa9804c810
I20250114 02:54:33.906649 1992 ts_tablet_manager.cc:1939] T 3161f75bc4e84000af022faa9804c810 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:33.923717 1207 tablet_bootstrap.cc:492] T 3161f75bc4e84000af022faa9804c810 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:33.924237 1207 ts_tablet_manager.cc:1397] T 3161f75bc4e84000af022faa9804c810 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.083s user 0.008s sys 0.002s
I20250114 02:54:33.928951 1207 raft_consensus.cc:357] T 3161f75bc4e84000af022faa9804c810 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:33.929698 1207 raft_consensus.cc:383] T 3161f75bc4e84000af022faa9804c810 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:33.929987 1207 raft_consensus.cc:738] T 3161f75bc4e84000af022faa9804c810 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:33.930241 1205 tablet_bootstrap.cc:654] T f40c66314342453b898da3ac0e7e86bf P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:33.934653 2247 consensus_queue.cc:1035] T 97f4e061d3b94ee9982410b7ce135453 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:33.959852 1207 consensus_queue.cc:260] T 3161f75bc4e84000af022faa9804c810 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:33.960953 2247 consensus_queue.cc:1035] T 97f4e061d3b94ee9982410b7ce135453 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:33.891871 1206 consensus_queue.cc:260] T be7aad41ccdd4ef9bcfe8a985155708f P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:33.979059 1207 ts_tablet_manager.cc:1428] T 3161f75bc4e84000af022faa9804c810 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.052s user 0.005s sys 0.000s
I20250114 02:54:33.983448 1206 ts_tablet_manager.cc:1428] T be7aad41ccdd4ef9bcfe8a985155708f P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.100s user 0.006s sys 0.000s
I20250114 02:54:33.984704 1206 tablet_bootstrap.cc:492] T a2d68eccaaa64511a9ae36f2b07a7b59 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:33.986929 1207 tablet_bootstrap.cc:492] T 6e2fb05f72b743e2a02ed36b63f80b23 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:33.992712 1992 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 290) took 3794 ms. Trace:
I20250114 02:54:33.993016 1992 rpcz_store.cc:276] 0114 02:54:30.197980 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:30.198074 (+ 94us) service_pool.cc:225] Handling call
0114 02:54:33.578903 (+3380829us) ts_tablet_manager.cc:1145] Deleting tablet 3161f75bc4e84000af022faa9804c810
0114 02:54:33.578953 (+ 50us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:33.845974 (+267021us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:33.905411 (+ 59437us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:33.992650 (+ 87239us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3380270}
I20250114 02:54:33.994326 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet 3161f75bc4e84000af022faa9804c810 (table default.Table_35a78676616e4b20906b262c9d0c0d9e [id=e984149e9da444f1b2802e3f35a60fd6]) successfully deleted
I20250114 02:54:34.007823 1207 tablet_bootstrap.cc:654] T 6e2fb05f72b743e2a02ed36b63f80b23 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:34.017182 1992 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 291) took 3711 ms. Trace:
I20250114 02:54:34.017642 1992 rpcz_store.cc:276] 0114 02:54:30.305565 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:30.306200 (+ 635us) service_pool.cc:225] Handling call
0114 02:54:34.016764 (+3710564us) ts_tablet_manager.cc:1145] Deleting tablet 25bce689024f49a49496e415efe91fdc
0114 02:54:34.016822 (+ 58us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:34.017119 (+ 297us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3709445}
W20250114 02:54:34.019555 458 catalog_manager.cc:4852] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet 25bce689024f49a49496e415efe91fdc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 25bce689024f49a49496e415efe91fdc already in progress: creating tablet
I20250114 02:54:34.020540 1992 tablet_replica.cc:331] T 99b940d872ce4781907517c40fe08781 P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:34.021461 1992 raft_consensus.cc:2238] T 99b940d872ce4781907517c40fe08781 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 LEADER]: Raft consensus shutting down.
I20250114 02:54:34.022589 1207 tablet_bootstrap.cc:492] T 6e2fb05f72b743e2a02ed36b63f80b23 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:34.023175 1207 ts_tablet_manager.cc:1397] T 6e2fb05f72b743e2a02ed36b63f80b23 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.036s user 0.007s sys 0.003s
I20250114 02:54:34.029486 1207 raft_consensus.cc:357] T 6e2fb05f72b743e2a02ed36b63f80b23 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:34.030146 1207 raft_consensus.cc:383] T 6e2fb05f72b743e2a02ed36b63f80b23 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:34.032532 1207 raft_consensus.cc:738] T 6e2fb05f72b743e2a02ed36b63f80b23 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:34.033129 1207 consensus_queue.cc:260] T 6e2fb05f72b743e2a02ed36b63f80b23 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:34.046828 1207 ts_tablet_manager.cc:1428] T 6e2fb05f72b743e2a02ed36b63f80b23 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.023s user 0.005s sys 0.000s
I20250114 02:54:34.047714 1207 tablet_bootstrap.cc:492] T 141cede72b8741a8b0615d7c6bb823bb P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:34.053134 1207 tablet_bootstrap.cc:654] T 141cede72b8741a8b0615d7c6bb823bb P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:34.062623 1205 tablet_bootstrap.cc:492] T f40c66314342453b898da3ac0e7e86bf P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:34.063144 1205 ts_tablet_manager.cc:1397] T f40c66314342453b898da3ac0e7e86bf P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.194s user 0.011s sys 0.000s
I20250114 02:54:34.065287 1205 raft_consensus.cc:357] T f40c66314342453b898da3ac0e7e86bf P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:34.065989 1205 raft_consensus.cc:383] T f40c66314342453b898da3ac0e7e86bf P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:34.066256 1205 raft_consensus.cc:738] T f40c66314342453b898da3ac0e7e86bf P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:34.066941 1205 consensus_queue.cc:260] T f40c66314342453b898da3ac0e7e86bf P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:34.073616 1207 tablet_bootstrap.cc:492] T 141cede72b8741a8b0615d7c6bb823bb P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:34.074029 1207 ts_tablet_manager.cc:1397] T 141cede72b8741a8b0615d7c6bb823bb P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.026s user 0.011s sys 0.001s
I20250114 02:54:34.077704 1207 raft_consensus.cc:357] T 141cede72b8741a8b0615d7c6bb823bb P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:34.080485 1207 raft_consensus.cc:383] T 141cede72b8741a8b0615d7c6bb823bb P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:34.080915 1207 raft_consensus.cc:738] T 141cede72b8741a8b0615d7c6bb823bb P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:34.081503 1207 consensus_queue.cc:260] T 141cede72b8741a8b0615d7c6bb823bb P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:34.087110 1207 ts_tablet_manager.cc:1428] T 141cede72b8741a8b0615d7c6bb823bb P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.013s user 0.004s sys 0.002s
I20250114 02:54:34.088002 1207 tablet_bootstrap.cc:492] T 30a56fb2d941470590e17f37b90aef26 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:34.101116 1084 tablet_service.cc:1467] Processing CreateTablet for tablet 5c26a33248d34bd2bfa0455249fa8156 (DEFAULT_TABLE table=default.Table_7416825652064407ab1bc5469bc21167 [id=17a03e2b5b374954963af7fe597dbced]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:34.102141 1084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c26a33248d34bd2bfa0455249fa8156. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:34.104198 2389 raft_consensus.cc:491] T 141cede72b8741a8b0615d7c6bb823bb P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:34.104671 2389 raft_consensus.cc:513] T 141cede72b8741a8b0615d7c6bb823bb P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:34.105393 1207 tablet_bootstrap.cc:654] T 30a56fb2d941470590e17f37b90aef26 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:34.106499 2389 leader_election.cc:290] T 141cede72b8741a8b0615d7c6bb823bb P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:34.109707 957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141cede72b8741a8b0615d7c6bb823bb" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:34.110311 957 raft_consensus.cc:2463] T 141cede72b8741a8b0615d7c6bb823bb P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 0.
I20250114 02:54:34.111511 1040 leader_election.cc:304] T 141cede72b8741a8b0615d7c6bb823bb P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:34.112326 2389 raft_consensus.cc:2798] T 141cede72b8741a8b0615d7c6bb823bb P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:34.112618 2389 raft_consensus.cc:491] T 141cede72b8741a8b0615d7c6bb823bb P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:34.112882 2389 raft_consensus.cc:3054] T 141cede72b8741a8b0615d7c6bb823bb P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:34.129736 1992 raft_consensus.cc:2267] T 99b940d872ce4781907517c40fe08781 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:34.124189 2389 raft_consensus.cc:513] T 141cede72b8741a8b0615d7c6bb823bb P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:34.131781 1992 ts_tablet_manager.cc:1905] T 99b940d872ce4781907517c40fe08781 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:34.135136 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141cede72b8741a8b0615d7c6bb823bb" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:34.135655 818 raft_consensus.cc:3054] T 141cede72b8741a8b0615d7c6bb823bb P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:34.138162 2389 leader_election.cc:290] T 141cede72b8741a8b0615d7c6bb823bb P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Requested vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:34.139921 1205 ts_tablet_manager.cc:1428] T f40c66314342453b898da3ac0e7e86bf P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.076s user 0.004s sys 0.000s
I20250114 02:54:34.142823 939 tablet_service.cc:1467] Processing CreateTablet for tablet 5c26a33248d34bd2bfa0455249fa8156 (DEFAULT_TABLE table=default.Table_7416825652064407ab1bc5469bc21167 [id=17a03e2b5b374954963af7fe597dbced]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:34.089370 1206 tablet_bootstrap.cc:654] T a2d68eccaaa64511a9ae36f2b07a7b59 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:34.108063 813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141cede72b8741a8b0615d7c6bb823bb" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:34.157972 1207 tablet_bootstrap.cc:492] T 30a56fb2d941470590e17f37b90aef26 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:34.158478 818 raft_consensus.cc:2463] T 141cede72b8741a8b0615d7c6bb823bb P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1.
I20250114 02:54:34.158572 1207 ts_tablet_manager.cc:1397] T 30a56fb2d941470590e17f37b90aef26 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.071s user 0.008s sys 0.006s
I20250114 02:54:34.159509 1042 leader_election.cc:304] T 141cede72b8741a8b0615d7c6bb823bb P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:34.163863 939 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c26a33248d34bd2bfa0455249fa8156. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:34.163951 1207 raft_consensus.cc:357] T 30a56fb2d941470590e17f37b90aef26 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:34.164593 1207 raft_consensus.cc:383] T 30a56fb2d941470590e17f37b90aef26 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:34.164825 1207 raft_consensus.cc:738] T 30a56fb2d941470590e17f37b90aef26 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:34.168622 2372 raft_consensus.cc:491] T 612987f3e4ea4a02932ceefa0d740425 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:34.168004 1207 consensus_queue.cc:260] T 30a56fb2d941470590e17f37b90aef26 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:34.169090 2372 raft_consensus.cc:513] T 612987f3e4ea4a02932ceefa0d740425 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:34.170890 2372 leader_election.cc:290] T 612987f3e4ea4a02932ceefa0d740425 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:34.171600 1207 ts_tablet_manager.cc:1428] T 30a56fb2d941470590e17f37b90aef26 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20250114 02:54:34.171732 1205 tablet_bootstrap.cc:492] T d2a7b2bee5c1440fb705b172c18a9247 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:34.162166 2389 raft_consensus.cc:2798] T 141cede72b8741a8b0615d7c6bb823bb P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:34.172636 957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "612987f3e4ea4a02932ceefa0d740425" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
W20250114 02:54:34.173827 751 leader_election.cc:343] T 612987f3e4ea4a02932ceefa0d740425 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:34.174850 2389 raft_consensus.cc:695] T 141cede72b8741a8b0615d7c6bb823bb P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Becoming Leader. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Running, Role: LEADER
I20250114 02:54:34.177654 1207 tablet_bootstrap.cc:492] T 967f4d5b944543d49e87645d069b5b55 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:34.177229 2389 consensus_queue.cc:237] T 141cede72b8741a8b0615d7c6bb823bb P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:34.181591 2410 tablet.cc:1721] T edc69b66bea54625a4b15867b75986b7 P 525198593c8e47afaee54e74267b87e3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250114 02:54:34.183712 2411 tablet.cc:1721] T edc69b66bea54625a4b15867b75986b7 P 819f9e3412e54d44abb11d8b3ca877c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250114 02:54:34.188934 786 tablet_service.cc:1467] Processing CreateTablet for tablet 5c26a33248d34bd2bfa0455249fa8156 (DEFAULT_TABLE table=default.Table_7416825652064407ab1bc5469bc21167 [id=17a03e2b5b374954963af7fe597dbced]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:34.189992 786 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c26a33248d34bd2bfa0455249fa8156. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:34.190001 462 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet edc69b66bea54625a4b15867b75986b7 (table default.Table_0a0bda55a4a9476c98d961822b8a19bd [id=fb2be5c1ca9444a197f3315ea9c59b09]). Expected version 1 got 0
I20250114 02:54:34.195019 957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141cede72b8741a8b0615d7c6bb823bb" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
I20250114 02:54:34.195606 957 raft_consensus.cc:3054] T 141cede72b8741a8b0615d7c6bb823bb P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:34.199426 932 tablet_service.cc:1467] Processing CreateTablet for tablet caf6857e3a1e4d2f8625526d0927311a (DEFAULT_TABLE table=default.Table_5623f4fcb2fb42ef8cf628812939a998 [id=0f4fef7ce07a494a811442a3762b0443]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:34.200512 932 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet caf6857e3a1e4d2f8625526d0927311a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:34.201907 957 raft_consensus.cc:2463] T 141cede72b8741a8b0615d7c6bb823bb P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1.
I20250114 02:54:34.203787 462 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 141cede72b8741a8b0615d7c6bb823bb (table default.Table_4364d1df16d149d2ba10469ce7288d4e [id=13fc996d911b430290e37a94dd44b1ab]). Expected version 1 got 0
I20250114 02:54:34.203902 1075 tablet_service.cc:1467] Processing CreateTablet for tablet caf6857e3a1e4d2f8625526d0927311a (DEFAULT_TABLE table=default.Table_5623f4fcb2fb42ef8cf628812939a998 [id=0f4fef7ce07a494a811442a3762b0443]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:34.204391 462 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 30a56fb2d941470590e17f37b90aef26 (table default.Table_0e0710e3f43e4e859ad32662704ee41c [id=66082998119e4c43a666fb43087fc622]). Expected version 1 got 0
I20250114 02:54:34.204993 1075 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet caf6857e3a1e4d2f8625526d0927311a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:34.204908 462 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 38fe8829d98547fb907ef78099df3d7f (table default.Table_60450873174a4c80aa7e2122343d82be [id=9299683349354f879847b7c9d93f6072]). Expected version 1 got 0
I20250114 02:54:34.204975 797 tablet_service.cc:1467] Processing CreateTablet for tablet caf6857e3a1e4d2f8625526d0927311a (DEFAULT_TABLE table=default.Table_5623f4fcb2fb42ef8cf628812939a998 [id=0f4fef7ce07a494a811442a3762b0443]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:34.205973 797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet caf6857e3a1e4d2f8625526d0927311a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:34.207895 462 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet a52d6749ff834b62a8e20a2f45935240 (table default.Table_1c9b7896941e4289b1c12e20c210c2c1 [id=b3ee15b9f10f4d489edbde3af9034d25]). Expected version 1 got 0
I20250114 02:54:34.209020 2097 raft_consensus.cc:491] T c86562688d5b4fffb71cbdf7be0afc37 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:34.209576 2097 raft_consensus.cc:513] T c86562688d5b4fffb71cbdf7be0afc37 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:34.211686 2097 leader_election.cc:290] T c86562688d5b4fffb71cbdf7be0afc37 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:34.215437 1099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "612987f3e4ea4a02932ceefa0d740425" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:34.217032 1103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c86562688d5b4fffb71cbdf7be0afc37" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:34.217679 1103 raft_consensus.cc:2463] T c86562688d5b4fffb71cbdf7be0afc37 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:34.219120 898 leader_election.cc:304] T c86562688d5b4fffb71cbdf7be0afc37 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters:
W20250114 02:54:34.220093 753 leader_election.cc:343] T 612987f3e4ea4a02932ceefa0d740425 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:34.220494 753 leader_election.cc:304] T 612987f3e4ea4a02932ceefa0d740425 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:34.220552 2247 raft_consensus.cc:2798] T c86562688d5b4fffb71cbdf7be0afc37 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:54:34.220863 2247 raft_consensus.cc:491] T c86562688d5b4fffb71cbdf7be0afc37 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:34.221189 2247 raft_consensus.cc:3054] T c86562688d5b4fffb71cbdf7be0afc37 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:54:34.221343 2372 raft_consensus.cc:2743] T 612987f3e4ea4a02932ceefa0d740425 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250114 02:54:34.224869 898 leader_election.cc:343] T c86562688d5b4fffb71cbdf7be0afc37 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Not found: Tablet not found: c86562688d5b4fffb71cbdf7be0afc37
I20250114 02:54:34.226131 1206 tablet_bootstrap.cc:492] T a2d68eccaaa64511a9ae36f2b07a7b59 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:34.226516 1206 ts_tablet_manager.cc:1397] T a2d68eccaaa64511a9ae36f2b07a7b59 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.242s user 0.004s sys 0.006s
I20250114 02:54:34.229090 2247 raft_consensus.cc:513] T c86562688d5b4fffb71cbdf7be0afc37 P 525198593c8e47afaee54e74267b87e3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:34.231504 1099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c86562688d5b4fffb71cbdf7be0afc37" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:34.232000 1099 raft_consensus.cc:3054] T c86562688d5b4fffb71cbdf7be0afc37 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:54:34.232312 2247 leader_election.cc:290] T c86562688d5b4fffb71cbdf7be0afc37 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 2 election: Requested vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
W20250114 02:54:34.233240 898 leader_election.cc:343] T c86562688d5b4fffb71cbdf7be0afc37 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Not found: Tablet not found: c86562688d5b4fffb71cbdf7be0afc37
I20250114 02:54:34.239341 1205 tablet_bootstrap.cc:654] T d2a7b2bee5c1440fb705b172c18a9247 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:34.239326 1099 raft_consensus.cc:2463] T c86562688d5b4fffb71cbdf7be0afc37 P 90d6c2c58bb54e0eafb365625ac33871 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 2.
I20250114 02:54:34.239519 462 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 719188f639724851a5fc2b37cb60fdbb (table default.Table_8f8a18d7868c44bbb4d66a0692b08186 [id=e6292728826f437cb75c524a8aef2e87]). Expected version 1 got 0
I20250114 02:54:34.240546 462 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet df045d80d8ce4ff194a6d2039a44bce6 (table default.Table_2befe573a994473abe0556ba7e713686 [id=da0258578bb34ae29551fc593bf7d0b5]). Expected version 1 got 0
I20250114 02:54:34.247285 1207 tablet_bootstrap.cc:654] T 967f4d5b944543d49e87645d069b5b55 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:34.250880 2372 raft_consensus.cc:491] T a1cf92e5275b454ca893b12f2d90d25b P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 525198593c8e47afaee54e74267b87e3)
I20250114 02:54:34.251317 2372 raft_consensus.cc:513] T a1cf92e5275b454ca893b12f2d90d25b P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:34.241984 1206 raft_consensus.cc:357] T a2d68eccaaa64511a9ae36f2b07a7b59 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:34.252597 1206 raft_consensus.cc:383] T a2d68eccaaa64511a9ae36f2b07a7b59 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:34.252840 1206 raft_consensus.cc:738] T a2d68eccaaa64511a9ae36f2b07a7b59 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:34.253034 2372 leader_election.cc:290] T a1cf92e5275b454ca893b12f2d90d25b P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:34.253407 1206 consensus_queue.cc:260] T a2d68eccaaa64511a9ae36f2b07a7b59 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:34.256083 1206 ts_tablet_manager.cc:1428] T a2d68eccaaa64511a9ae36f2b07a7b59 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.029s user 0.005s sys 0.001s
W20250114 02:54:34.259186 753 leader_election.cc:343] T a1cf92e5275b454ca893b12f2d90d25b P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Not found: Tablet not found: a1cf92e5275b454ca893b12f2d90d25b
I20250114 02:54:34.267362 2389 raft_consensus.cc:491] T 3161f75bc4e84000af022faa9804c810 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:34.268124 2375 raft_consensus.cc:491] T a2d68eccaaa64511a9ae36f2b07a7b59 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:34.267848 2389 raft_consensus.cc:513] T 3161f75bc4e84000af022faa9804c810 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:34.268582 2375 raft_consensus.cc:513] T a2d68eccaaa64511a9ae36f2b07a7b59 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:34.269902 2389 leader_election.cc:290] T 3161f75bc4e84000af022faa9804c810 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:34.270305 2375 leader_election.cc:290] T a2d68eccaaa64511a9ae36f2b07a7b59 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
W20250114 02:54:34.272787 1042 leader_election.cc:343] T 3161f75bc4e84000af022faa9804c810 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Not found: Tablet not found: 3161f75bc4e84000af022faa9804c810
W20250114 02:54:34.276063 1040 leader_election.cc:343] T 3161f75bc4e84000af022faa9804c810 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Not found: Tablet not found: 3161f75bc4e84000af022faa9804c810
I20250114 02:54:34.276468 1040 leader_election.cc:304] T 3161f75bc4e84000af022faa9804c810 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90d6c2c58bb54e0eafb365625ac33871; no voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2
I20250114 02:54:34.276804 1100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2d68eccaaa64511a9ae36f2b07a7b59" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:34.277312 2389 raft_consensus.cc:2743] T 3161f75bc4e84000af022faa9804c810 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250114 02:54:34.277840 753 leader_election.cc:343] T a2d68eccaaa64511a9ae36f2b07a7b59 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:34.282706 464 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: default.Table_0139fab1a81440309d9d7729c3e76c8a [id=2688bb1e2f4f4765986f738256d17f78] alter complete (version 1)
I20250114 02:54:34.240270 898 leader_election.cc:304] T c86562688d5b4fffb71cbdf7be0afc37 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters: 819f9e3412e54d44abb11d8b3ca877c2
W20250114 02:54:34.319579 898 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.67:45749, user_credentials={real_user=slave}} blocked reactor thread for 79419us
I20250114 02:54:34.320988 2097 raft_consensus.cc:2798] T c86562688d5b4fffb71cbdf7be0afc37 P 525198593c8e47afaee54e74267b87e3 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 02:54:34.321408 2097 raft_consensus.cc:695] T c86562688d5b4fffb71cbdf7be0afc37 P 525198593c8e47afaee54e74267b87e3 [term 2 LEADER]: Becoming Leader. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Running, Role: LEADER
I20250114 02:54:34.322111 2097 consensus_queue.cc:237] T c86562688d5b4fffb71cbdf7be0afc37 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:34.323686 2389 raft_consensus.cc:491] T ed7f575653c848d7b076c17c38c4a244 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:34.324095 2389 raft_consensus.cc:513] T ed7f575653c848d7b076c17c38c4a244 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:34.325327 1100 raft_consensus.cc:3054] T 30a56fb2d941470590e17f37b90aef26 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:34.326277 2389 leader_election.cc:290] T ed7f575653c848d7b076c17c38c4a244 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:34.329005 467 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: default.Table_428bd60f16824ac2b8bfaad6819f76e1 [id=35c5879da4b344118a377335fc6f83de] alter complete (version 1)
I20250114 02:54:34.329619 957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2d68eccaaa64511a9ae36f2b07a7b59" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:34.330564 959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7f575653c848d7b076c17c38c4a244" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:34.331118 959 raft_consensus.cc:2463] T ed7f575653c848d7b076c17c38c4a244 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 0.
I20250114 02:54:34.332207 1040 leader_election.cc:304] T ed7f575653c848d7b076c17c38c4a244 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:34.333770 2389 raft_consensus.cc:2798] T ed7f575653c848d7b076c17c38c4a244 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:34.334116 2389 raft_consensus.cc:491] T ed7f575653c848d7b076c17c38c4a244 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:34.334447 2389 raft_consensus.cc:3054] T ed7f575653c848d7b076c17c38c4a244 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:34.336769 1075 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ac7708550954a209e2a8d217c3b39bd with delete_type TABLET_DATA_DELETED (replaced by tablet d0cc8b9e3aee4292bc69e8053d7e8cc1) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:34.337841 1076 tablet_service.cc:1514] Processing DeleteTablet for tablet 1381e26725174b9c877075cf5444f566 with delete_type TABLET_DATA_DELETED (replaced by tablet 9ffec1d1bb594fda83df8d94d4533d56) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:34.338358 1085 tablet_service.cc:1514] Processing DeleteTablet for tablet 84738a6c78b24c1fb64013048763b18f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:23 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:34.338708 1081 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e858a8449444ef999b9bbef612249d3 with delete_type TABLET_DATA_DELETED (replaced by tablet 797d6cdf36704c98aa106aac87a945ca) from {username='slave'} at 127.0.0.1:53876
W20250114 02:54:34.339419 1042 leader_election.cc:343] T ed7f575653c848d7b076c17c38c4a244 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Not found: Tablet not found: ed7f575653c848d7b076c17c38c4a244
I20250114 02:54:34.338349 1086 tablet_service.cc:1514] Processing DeleteTablet for tablet 179248cb5acf4740b268e0ec8b2852f9 with delete_type TABLET_DATA_DELETED (replaced by tablet efcb1bcd47174cde8c8ab7f99b3bf21e) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:34.342862 2375 raft_consensus.cc:491] T 670dc7ff15d644e18c296eeb2a6b99e1 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:34.343290 2375 raft_consensus.cc:513] T 670dc7ff15d644e18c296eeb2a6b99e1 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:34.345163 2375 leader_election.cc:290] T 670dc7ff15d644e18c296eeb2a6b99e1 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:34.347676 1099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "670dc7ff15d644e18c296eeb2a6b99e1" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
W20250114 02:54:34.348896 753 leader_election.cc:343] T 670dc7ff15d644e18c296eeb2a6b99e1 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:34.358485 467 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: default.Table_990b8d5f1bd146d6b358457f954780c9 [id=26627ccdc808451fb9593ad7a6df6991] alter complete (version 2)
I20250114 02:54:34.358981 2389 raft_consensus.cc:513] T ed7f575653c848d7b076c17c38c4a244 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
W20250114 02:54:34.362479 1042 leader_election.cc:343] T ed7f575653c848d7b076c17c38c4a244 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Not found: Tablet not found: ed7f575653c848d7b076c17c38c4a244
I20250114 02:54:34.363620 2389 leader_election.cc:290] T ed7f575653c848d7b076c17c38c4a244 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Requested vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:34.367741 1084 tablet_service.cc:1514] Processing DeleteTablet for tablet de4ca83a69704a368a25954cdc1b9392 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:25 UTC) from {username='slave'} at 127.0.0.1:53876
W20250114 02:54:34.368603 898 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.67:45749, user_credentials={real_user=slave}} blocked reactor thread for 46709.2us
W20250114 02:54:34.369411 751 leader_election.cc:343] T a1cf92e5275b454ca893b12f2d90d25b P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Not found: Tablet not found: a1cf92e5275b454ca893b12f2d90d25b
I20250114 02:54:34.369652 959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7f575653c848d7b076c17c38c4a244" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
I20250114 02:54:34.369781 751 leader_election.cc:304] T a1cf92e5275b454ca893b12f2d90d25b P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:34.370138 959 raft_consensus.cc:3054] T ed7f575653c848d7b076c17c38c4a244 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:34.370666 2375 raft_consensus.cc:2743] T a1cf92e5275b454ca893b12f2d90d25b P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250114 02:54:34.372960 751 leader_election.cc:343] T a2d68eccaaa64511a9ae36f2b07a7b59 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:34.373390 1081 tablet_service.cc:1514] Processing DeleteTablet for tablet e15e6e52d7294aeeb4463b3a76022ec3 with delete_type TABLET_DATA_DELETED (replaced by tablet 6d741bfd00784d2d830be3ebe50dc7c4) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:34.373346 751 leader_election.cc:304] T a2d68eccaaa64511a9ae36f2b07a7b59 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:34.373911 1086 tablet_service.cc:1514] Processing DeleteTablet for tablet 46360f726fad4534a2daf67581346fc1 with delete_type TABLET_DATA_DELETED (replaced by tablet be7aad41ccdd4ef9bcfe8a985155708f) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:34.374321 1085 tablet_service.cc:1514] Processing DeleteTablet for tablet d3fb28717c6f4844af84bb817fe7dfc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:25 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:34.374781 813 raft_consensus.cc:1270] T 719188f639724851a5fc2b37cb60fdbb P 819f9e3412e54d44abb11d8b3ca877c2 [term 2 FOLLOWER]: Refusing update from remote peer 525198593c8e47afaee54e74267b87e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250114 02:54:34.375329 2375 raft_consensus.cc:2743] T a2d68eccaaa64511a9ae36f2b07a7b59 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:34.378041 2143 consensus_queue.cc:1035] T 30a56fb2d941470590e17f37b90aef26 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:34.378443 957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "670dc7ff15d644e18c296eeb2a6b99e1" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:34.379695 934 tablet_service.cc:1514] Processing DeleteTablet for tablet f0bd9bd5a17d418d8082ed0ebd1bf4cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:19 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:34.380242 931 tablet_service.cc:1514] Processing DeleteTablet for tablet 25e85b0c5c9a4a3ca35a001359f21167 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:29 UTC) from {username='slave'} at 127.0.0.1:55068
W20250114 02:54:34.381600 751 leader_election.cc:343] T 670dc7ff15d644e18c296eeb2a6b99e1 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:34.382074 751 leader_election.cc:304] T 670dc7ff15d644e18c296eeb2a6b99e1 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:34.383510 2375 raft_consensus.cc:2743] T 670dc7ff15d644e18c296eeb2a6b99e1 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:34.384685 2143 consensus_queue.cc:1035] T 719188f639724851a5fc2b37cb60fdbb P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250114 02:54:34.385854 463 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.66:50663 (request call id 51) took 5218 ms (client timeout 1000 ms). Trace:
I20250114 02:54:34.385874 959 raft_consensus.cc:2463] T ed7f575653c848d7b076c17c38c4a244 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1.
W20250114 02:54:34.386186 463 rpcz_store.cc:269] 0114 02:54:29.166996 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:29.278618 (+111622us) service_pool.cc:225] Handling call
0114 02:54:29.547893 (+269275us) spinlock_profiling.cc:243] Waited 46.8 ms on lock 0x7b54000316d4. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616cad50 00007f646174df2a 00007f64633f2dec 00007f64634eadee 00007f6461248fc4 00007f6461248f52 00007f6461248e81 00007f6461248dfc 00007f64612480b2 00007f645f512dfc 00007f645f512236
0114 02:54:34.385782 (+4837889us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6476,"reported_tablets":30,"spinlock_wait_cycles":107679488}
W20250114 02:54:34.387490 463 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.66:50663 (request call id 55) took 1544 ms (client timeout 1000 ms). Trace:
W20250114 02:54:34.387779 463 rpcz_store.cc:269] 0114 02:54:32.842869 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:34.387227 (+1544358us) service_pool.cc:212] Skipping call since client already timed out
0114 02:54:34.387440 (+ 213us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250114 02:54:34.392520 935 tablet_service.cc:1514] Processing DeleteTablet for tablet e15e6e52d7294aeeb4463b3a76022ec3 with delete_type TABLET_DATA_DELETED (replaced by tablet 6d741bfd00784d2d830be3ebe50dc7c4) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:34.393061 936 tablet_service.cc:1514] Processing DeleteTablet for tablet c86562688d5b4fffb71cbdf7be0afc37 with delete_type TABLET_DATA_DELETED (replaced by tablet e7ca0bd1dbd542de9ee11e6c72522ae0) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:34.394578 936 tablet_service.cc:1514] Processing DeleteTablet for tablet 99b940d872ce4781907517c40fe08781 with delete_type TABLET_DATA_DELETED (replaced by tablet fbeddf984d1f466192f60033715855ad) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:34.396615 936 tablet_service.cc:1514] Processing DeleteTablet for tablet 84738a6c78b24c1fb64013048763b18f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:23 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:34.398006 1040 leader_election.cc:304] T ed7f575653c848d7b076c17c38c4a244 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters: 819f9e3412e54d44abb11d8b3ca877c2
I20250114 02:54:34.398871 2389 raft_consensus.cc:2798] T ed7f575653c848d7b076c17c38c4a244 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:34.399361 2389 raft_consensus.cc:695] T ed7f575653c848d7b076c17c38c4a244 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Becoming Leader. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Running, Role: LEADER
I20250114 02:54:34.400455 2375 raft_consensus.cc:491] T 0b8361cd538046e0bcb4aac821748f67 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:34.400306 2389 consensus_queue.cc:237] T ed7f575653c848d7b076c17c38c4a244 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:34.400978 2375 raft_consensus.cc:513] T 0b8361cd538046e0bcb4aac821748f67 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:34.402348 1084 tablet_service.cc:1514] Processing DeleteTablet for tablet 25e85b0c5c9a4a3ca35a001359f21167 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:29 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:34.403378 2375 leader_election.cc:290] T 0b8361cd538046e0bcb4aac821748f67 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:34.408347 957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b8361cd538046e0bcb4aac821748f67" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:34.409054 939 tablet_service.cc:1514] Processing DeleteTablet for tablet de4ca83a69704a368a25954cdc1b9392 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:25 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:34.409622 468 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: default.Table_0518d8bed7304c86aff32e05e6f07c0d [id=515a2fe3f1e84e82bccc02ef5cecef1b] alter complete (version 1)
W20250114 02:54:34.409781 751 leader_election.cc:343] T 0b8361cd538046e0bcb4aac821748f67 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:34.410631 2419 tablet.cc:1721] T edc69b66bea54625a4b15867b75986b7 P 90d6c2c58bb54e0eafb365625ac33871: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250114 02:54:34.411599 470 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.67:34825 (request call id 51) took 4908 ms (client timeout 1000 ms). Trace:
W20250114 02:54:34.412019 470 rpcz_store.cc:269] 0114 02:54:29.503212 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:29.556017 (+ 52805us) service_pool.cc:225] Handling call
0114 02:54:34.411535 (+4855518us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26000,"reported_tablets":35}
W20250114 02:54:34.413518 470 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.67:34825 (request call id 54) took 1287 ms (client timeout 1000 ms). Trace:
I20250114 02:54:34.414831 1207 tablet_bootstrap.cc:492] T 967f4d5b944543d49e87645d069b5b55 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:34.415294 1207 ts_tablet_manager.cc:1397] T 967f4d5b944543d49e87645d069b5b55 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.238s user 0.018s sys 0.013s
W20250114 02:54:34.415694 470 rpcz_store.cc:269] 0114 02:54:33.125826 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:34.413385 (+1287559us) service_pool.cc:212] Skipping call since client already timed out
0114 02:54:34.413471 (+ 86us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250114 02:54:34.419502 2375 raft_consensus.cc:491] T 2837990edc4d4bea871a8114533c9352 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 90d6c2c58bb54e0eafb365625ac33871)
I20250114 02:54:34.420125 2375 raft_consensus.cc:513] T 2837990edc4d4bea871a8114533c9352 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:34.423691 2375 leader_election.cc:290] T 2837990edc4d4bea871a8114533c9352 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
W20250114 02:54:34.425469 753 leader_election.cc:343] T 2837990edc4d4bea871a8114533c9352 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Not found: Tablet not found: 2837990edc4d4bea871a8114533c9352
I20250114 02:54:34.428699 2375 raft_consensus.cc:491] T be7aad41ccdd4ef9bcfe8a985155708f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:34.428797 1104 raft_consensus.cc:1270] T 719188f639724851a5fc2b37cb60fdbb P 90d6c2c58bb54e0eafb365625ac33871 [term 2 FOLLOWER]: Refusing update from remote peer 525198593c8e47afaee54e74267b87e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250114 02:54:34.419127 465 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: default.Table_9da872c9a22e4d07947df8db0524e61f [id=1f964e9ed159449580fe5326673c18ab] alter complete (version 1)
I20250114 02:54:34.429116 2375 raft_consensus.cc:513] T be7aad41ccdd4ef9bcfe8a985155708f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:34.432211 1104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be7aad41ccdd4ef9bcfe8a985155708f" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
W20250114 02:54:34.433236 753 leader_election.cc:343] T be7aad41ccdd4ef9bcfe8a985155708f P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:34.432004 2375 leader_election.cc:290] T be7aad41ccdd4ef9bcfe8a985155708f P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
W20250114 02:54:34.440753 751 leader_election.cc:343] T 2837990edc4d4bea871a8114533c9352 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Not found: Tablet not found: 2837990edc4d4bea871a8114533c9352
I20250114 02:54:34.440696 786 tablet_service.cc:1514] Processing DeleteTablet for tablet de4ca83a69704a368a25954cdc1b9392 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:25 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:34.441152 751 leader_election.cc:304] T 2837990edc4d4bea871a8114533c9352 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:34.441404 959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be7aad41ccdd4ef9bcfe8a985155708f" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:34.442004 2375 raft_consensus.cc:2743] T 2837990edc4d4bea871a8114533c9352 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:54:34.443411 1099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b8361cd538046e0bcb4aac821748f67" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:34.443890 1206 tablet_bootstrap.cc:492] T d56255ca806d4e2cac09fbb5e1b1efbd P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
W20250114 02:54:34.444478 753 leader_election.cc:343] T 0b8361cd538046e0bcb4aac821748f67 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:34.444801 753 leader_election.cc:304] T 0b8361cd538046e0bcb4aac821748f67 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:34.445683 2375 raft_consensus.cc:2743] T 0b8361cd538046e0bcb4aac821748f67 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:34.438241 1207 raft_consensus.cc:357] T 967f4d5b944543d49e87645d069b5b55 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:34.447981 1207 raft_consensus.cc:383] T 967f4d5b944543d49e87645d069b5b55 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:34.448390 1207 raft_consensus.cc:738] T 967f4d5b944543d49e87645d069b5b55 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
W20250114 02:54:34.450088 751 leader_election.cc:343] T be7aad41ccdd4ef9bcfe8a985155708f P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:34.450011 2247 consensus_queue.cc:1035] T 719188f639724851a5fc2b37cb60fdbb P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:34.450431 751 leader_election.cc:304] T be7aad41ccdd4ef9bcfe8a985155708f P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:34.451253 2372 raft_consensus.cc:2743] T be7aad41ccdd4ef9bcfe8a985155708f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:34.451917 786 tablet_service.cc:1514] Processing DeleteTablet for tablet e15e6e52d7294aeeb4463b3a76022ec3 with delete_type TABLET_DATA_DELETED (replaced by tablet 6d741bfd00784d2d830be3ebe50dc7c4) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:34.452950 785 tablet_service.cc:1514] Processing DeleteTablet for tablet ed7f575653c848d7b076c17c38c4a244 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:26 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:34.454174 784 tablet_service.cc:1514] Processing DeleteTablet for tablet 1381e26725174b9c877075cf5444f566 with delete_type TABLET_DATA_DELETED (replaced by tablet 9ffec1d1bb594fda83df8d94d4533d56) from {username='slave'} at 127.0.0.1:53078
W20250114 02:54:34.455405 458 catalog_manager.cc:4945] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): alter failed for tablet 99b940d872ce4781907517c40fe08781 (table default.Table_62251807af5845a89e89f8b2e003928f [id=e3418787ef0a4e22979fb41d4f94ebb0]),no further retry: Illegal state: Tablet not RUNNING: STOPPED
I20250114 02:54:34.455533 786 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e858a8449444ef999b9bbef612249d3 with delete_type TABLET_DATA_DELETED (replaced by tablet 797d6cdf36704c98aa106aac87a945ca) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:34.457091 785 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ac7708550954a209e2a8d217c3b39bd with delete_type TABLET_DATA_DELETED (replaced by tablet d0cc8b9e3aee4292bc69e8053d7e8cc1) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:34.454231 797 tablet_service.cc:1514] Processing DeleteTablet for tablet 59a46bc865cc48df8e98598764fd1b54 with delete_type TABLET_DATA_DELETED (replaced by tablet cb8aa4212982417b861323c4809017d4) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:34.464287 932 tablet_service.cc:1514] Processing DeleteTablet for tablet ed7f575653c848d7b076c17c38c4a244 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:26 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:34.468554 797 tablet_service.cc:1514] Processing DeleteTablet for tablet c86562688d5b4fffb71cbdf7be0afc37 with delete_type TABLET_DATA_DELETED (replaced by tablet e7ca0bd1dbd542de9ee11e6c72522ae0) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:34.458776 1207 consensus_queue.cc:260] T 967f4d5b944543d49e87645d069b5b55 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:34.470449 1100 raft_consensus.cc:3054] T 6e2fb05f72b743e2a02ed36b63f80b23 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:34.471624 2418 tablet_replica.cc:331] T 6e858a8449444ef999b9bbef612249d3 P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
W20250114 02:54:34.472522 458 catalog_manager.cc:4839] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 5ac7708550954a209e2a8d217c3b39bd because the tablet was not found. No further retry: Not found: Tablet not found: 5ac7708550954a209e2a8d217c3b39bd
W20250114 02:54:34.473202 458 catalog_manager.cc:4839] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 1381e26725174b9c877075cf5444f566 because the tablet was not found. No further retry: Not found: Tablet not found: 1381e26725174b9c877075cf5444f566
W20250114 02:54:34.473697 458 catalog_manager.cc:4839] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 84738a6c78b24c1fb64013048763b18f because the tablet was not found. No further retry: Not found: Tablet not found: 84738a6c78b24c1fb64013048763b18f
I20250114 02:54:34.477459 2420 tablet_replica.cc:331] T c86562688d5b4fffb71cbdf7be0afc37 P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
W20250114 02:54:34.479032 456 catalog_manager.cc:4839] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet f0bd9bd5a17d418d8082ed0ebd1bf4cc because the tablet was not found. No further retry: Not found: Tablet not found: f0bd9bd5a17d418d8082ed0ebd1bf4cc
W20250114 02:54:34.479609 456 catalog_manager.cc:4839] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet 25e85b0c5c9a4a3ca35a001359f21167 because the tablet was not found. No further retry: Not found: Tablet not found: 25e85b0c5c9a4a3ca35a001359f21167
W20250114 02:54:34.480167 456 catalog_manager.cc:4839] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet e15e6e52d7294aeeb4463b3a76022ec3 because the tablet was not found. No further retry: Not found: Tablet not found: e15e6e52d7294aeeb4463b3a76022ec3
I20250114 02:54:34.487080 797 tablet_service.cc:1514] Processing DeleteTablet for tablet d3fb28717c6f4844af84bb817fe7dfc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:25 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:34.487182 1205 tablet_bootstrap.cc:492] T d2a7b2bee5c1440fb705b172c18a9247 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:34.487560 1205 ts_tablet_manager.cc:1397] T d2a7b2bee5c1440fb705b172c18a9247 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.316s user 0.011s sys 0.004s
I20250114 02:54:34.487816 2418 raft_consensus.cc:2238] T 6e858a8449444ef999b9bbef612249d3 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Raft consensus shutting down.
I20250114 02:54:34.489267 2418 raft_consensus.cc:2267] T 6e858a8449444ef999b9bbef612249d3 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:34.491197 2418 ts_tablet_manager.cc:1905] T 6e858a8449444ef999b9bbef612249d3 P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:34.492022 1075 tablet_service.cc:1514] Processing DeleteTablet for tablet 84738a6c78b24c1fb64013048763b18f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:23 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:34.493187 1205 raft_consensus.cc:357] T d2a7b2bee5c1440fb705b172c18a9247 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:34.493778 1205 raft_consensus.cc:383] T d2a7b2bee5c1440fb705b172c18a9247 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:34.494019 1205 raft_consensus.cc:738] T d2a7b2bee5c1440fb705b172c18a9247 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:34.494549 1205 consensus_queue.cc:260] T d2a7b2bee5c1440fb705b172c18a9247 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:34.481055 1207 ts_tablet_manager.cc:1428] T 967f4d5b944543d49e87645d069b5b55 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.065s user 0.006s sys 0.000s
I20250114 02:54:34.497542 1207 tablet_bootstrap.cc:492] T d2a7b2bee5c1440fb705b172c18a9247 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:34.499639 797 tablet_service.cc:1514] Processing DeleteTablet for tablet 97f4e061d3b94ee9982410b7ce135453 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:21 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:34.500844 2143 consensus_queue.cc:1035] T 30a56fb2d941470590e17f37b90aef26 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250114 02:54:34.502815 467 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.65:49805 (request call id 54) took 5381 ms (client timeout 1000 ms). Trace:
W20250114 02:54:34.503086 464 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.66:50663 (request call id 52) took 4081 ms (client timeout 1000 ms). Trace:
I20250114 02:54:34.505089 1205 ts_tablet_manager.cc:1428] T d2a7b2bee5c1440fb705b172c18a9247 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.017s user 0.006s sys 0.001s
I20250114 02:54:34.505529 797 tablet_service.cc:1514] Processing DeleteTablet for tablet 3161f75bc4e84000af022faa9804c810 with delete_type TABLET_DATA_DELETED (replaced by tablet 53929328fa51408aa98d7825018b6fdb) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:34.505965 1205 tablet_bootstrap.cc:492] T 0b8989d340fa4ae992ef97e226d41110 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:34.509452 932 tablet_service.cc:1514] Processing DeleteTablet for tablet e15e6e52d7294aeeb4463b3a76022ec3 with delete_type TABLET_DATA_DELETED (replaced by tablet 6d741bfd00784d2d830be3ebe50dc7c4) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:34.509505 934 tablet_service.cc:1514] Processing DeleteTablet for tablet 25e85b0c5c9a4a3ca35a001359f21167 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:29 UTC) from {username='slave'} at 127.0.0.1:55068
W20250114 02:54:34.503098 467 rpcz_store.cc:269] 0114 02:54:29.121588 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:29.193922 (+ 72334us) service_pool.cc:225] Handling call
0114 02:54:34.502754 (+5308832us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:30.860739 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:30.861172 (+ 433us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:30.861196 (+ 24us) write_op.cc:435] Acquired schema lock
0114 02:54:30.861212 (+ 16us) tablet.cc:659] Decoding operations
0114 02:54:30.861497 (+ 285us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:30.861540 (+ 43us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:30.861567 (+ 27us) tablet.cc:682] Acquiring locks for 1 operations
0114 02:54:30.861725 (+ 158us) tablet.cc:698] Row locks acquired
0114 02:54:30.861743 (+ 18us) write_op.cc:260] PREPARE: finished
0114 02:54:30.861864 (+ 121us) write_op.cc:270] Start()
0114 02:54:30.861954 (+ 90us) write_op.cc:276] Timestamp: P: 1736823270861786 usec, L: 0
0114 02:54:30.861975 (+ 21us) op_driver.cc:348] REPLICATION: starting
0114 02:54:30.862491 (+ 516us) log.cc:844] Serialized 356 byte log entry
0114 02:54:30.914046 (+ 51555us) op_driver.cc:464] REPLICATION: finished
0114 02:54:30.914509 (+ 463us) write_op.cc:301] APPLY: starting
0114 02:54:30.914568 (+ 59us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:30.914668 (+ 100us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:30.914774 (+ 106us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:30.914947 (+ 173us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:30.915104 (+ 157us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:30.915153 (+ 49us) write_op.cc:312] APPLY: finished
0114 02:54:30.915346 (+ 193us) log.cc:844] Serialized 23 byte log entry
0114 02:54:30.915490 (+ 144us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:30.915550 (+ 60us) write_op.cc:454] Released schema lock
0114 02:54:30.915598 (+ 48us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:30.915809 (+ 211us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:30.915831 (+ 22us) write_op.cc:454] Released schema lock
Related trace 'op':
0114 02:54:34.342948 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:34.343283 (+ 335us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:34.343318 (+ 35us) write_op.cc:435] Acquired schema lock
0114 02:54:34.343332 (+ 14us) tablet.cc:659] Decoding operations
0114 02:54:34.343572 (+ 240us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:34.343608 (+ 36us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:34.343631 (+ 23us) tablet.cc:682] Acquiring locks for 1 operations
0114 02:54:34.343793 (+ 162us) tablet.cc:698] Row locks acquired
0114 02:54:34.343805 (+ 12us) write_op.cc:260] PREPARE: finished
0114 02:54:34.343910 (+ 105us) write_op.cc:270] Start()
0114 02:54:34.343986 (+ 76us) write_op.cc:276] Timestamp: P: 1736823274343847 usec, L: 0
0114 02:54:34.344000 (+ 14us) op_driver.cc:348] REPLICATION: starting
0114 02:54:34.344596 (+ 596us) log.cc:844] Serialized 356 byte log entry
0114 02:54:34.354868 (+ 10272us) op_driver.cc:464] REPLICATION: finished
0114 02:54:34.355171 (+ 303us) write_op.cc:301] APPLY: starting
0114 02:54:34.355232 (+ 61us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:34.355343 (+ 111us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:34.355453 (+ 110us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:34.355658 (+ 205us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:34.355812 (+ 154us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:34.355867 (+ 55us) write_op.cc:312] APPLY: finished
0114 02:54:34.356069 (+ 202us) log.cc:844] Serialized 23 byte log entry
0114 02:54:34.356216 (+ 147us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:34.356310 (+ 94us) write_op.cc:454] Released schema lock
0114 02:54:34.356349 (+ 39us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:34.356585 (+ 236us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:34.356623 (+ 38us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":86150,"reported_tablets":32,"child_traces":[["op",{"apply.queue_time_us":284,"apply.run_cpu_time_us":1282,"apply.run_wall_time_us":1281,"num_ops":1,"prepare.queue_time_us":195372,"prepare.run_cpu_time_us":2111,"prepare.run_wall_time_us":6665,"raft.queue_time_us":142,"raft.run_cpu_time_us":603,"raft.run_wall_time_us":604,"replication_time_us":51972}],["op",{"apply.queue_time_us":160,"apply.run_cpu_time_us":1370,"apply.run_wall_time_us":1369,"num_ops":1,"prepare.queue_time_us":312,"prepare.run_cpu_time_us":1835,"prepare.run_wall_time_us":3092,"raft.queue_time_us":359,"raft.run_cpu_time_us":660,"raft.run_wall_time_us":673,"replication_time_us":10759}]]}
W20250114 02:54:34.521939 466 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.65:49805 (request call id 56) took 4180 ms (client timeout 1000 ms). Trace:
I20250114 02:54:34.525542 1075 tablet_service.cc:1514] Processing DeleteTablet for tablet c86562688d5b4fffb71cbdf7be0afc37 with delete_type TABLET_DATA_DELETED (replaced by tablet e7ca0bd1dbd542de9ee11e6c72522ae0) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:34.553680 2097 raft_consensus.cc:491] T f40c66314342453b898da3ac0e7e86bf P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:34.554265 2097 raft_consensus.cc:513] T f40c66314342453b898da3ac0e7e86bf P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:34.556450 2097 leader_election.cc:290] T f40c66314342453b898da3ac0e7e86bf P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:34.570843 1099 raft_consensus.cc:3054] T 967f4d5b944543d49e87645d069b5b55 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:34.573432 1103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40c66314342453b898da3ac0e7e86bf" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:34.560572 2420 raft_consensus.cc:2238] T c86562688d5b4fffb71cbdf7be0afc37 P 525198593c8e47afaee54e74267b87e3 [term 2 LEADER]: Raft consensus shutting down.
W20250114 02:54:34.576323 898 leader_election.cc:343] T f40c66314342453b898da3ac0e7e86bf P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:34.581471 2143 consensus_queue.cc:1035] T 967f4d5b944543d49e87645d069b5b55 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:34.597460 1206 tablet_bootstrap.cc:654] T d56255ca806d4e2cac09fbb5e1b1efbd P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
W20250114 02:54:34.601336 465 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.65:49805 (request call id 57) took 3161 ms (client timeout 1000 ms). Trace:
I20250114 02:54:34.604472 813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40c66314342453b898da3ac0e7e86bf" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
W20250114 02:54:34.604923 471 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.67:34825 (request call id 53) took 2659 ms (client timeout 1000 ms). Trace:
I20250114 02:54:34.605093 813 raft_consensus.cc:2463] T f40c66314342453b898da3ac0e7e86bf P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:34.606566 898 leader_election.cc:304] T f40c66314342453b898da3ac0e7e86bf P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters: 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:34.607257 2143 raft_consensus.cc:2798] T f40c66314342453b898da3ac0e7e86bf P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:34.607597 2143 raft_consensus.cc:491] T f40c66314342453b898da3ac0e7e86bf P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250114 02:54:34.605250 471 rpcz_store.cc:269] 0114 02:54:31.945067 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:32.050176 (+105109us) service_pool.cc:225] Handling call
0114 02:54:34.604844 (+2554668us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:33.716517 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:33.716877 (+ 360us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:33.716901 (+ 24us) write_op.cc:435] Acquired schema lock
0114 02:54:33.716914 (+ 13us) tablet.cc:659] Decoding operations
0114 02:54:33.717158 (+ 244us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:33.717195 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:33.717216 (+ 21us) tablet.cc:682] Acquiring locks for 1 operations
0114 02:54:33.717353 (+ 137us) tablet.cc:698] Row locks acquired
0114 02:54:33.717365 (+ 12us) write_op.cc:260] PREPARE: finished
0114 02:54:33.717516 (+ 151us) write_op.cc:270] Start()
0114 02:54:33.717598 (+ 82us) write_op.cc:276] Timestamp: P: 1736823273717443 usec, L: 0
0114 02:54:33.717613 (+ 15us) op_driver.cc:348] REPLICATION: starting
0114 02:54:33.718052 (+ 439us) log.cc:844] Serialized 530 byte log entry
0114 02:54:33.737264 (+ 19212us) op_driver.cc:464] REPLICATION: finished
0114 02:54:33.737687 (+ 423us) write_op.cc:301] APPLY: starting
0114 02:54:33.737738 (+ 51us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:33.737836 (+ 98us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:33.737930 (+ 94us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:33.738135 (+ 205us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:33.738266 (+ 131us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:33.738306 (+ 40us) write_op.cc:312] APPLY: finished
0114 02:54:33.738506 (+ 200us) log.cc:844] Serialized 23 byte log entry
0114 02:54:33.738638 (+ 132us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:33.738705 (+ 67us) write_op.cc:454] Released schema lock
0114 02:54:33.738734 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:33.738982 (+ 248us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:33.739003 (+ 21us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2720,"reported_tablets":32,"child_traces":[["op",{"apply.queue_time_us":277,"apply.run_cpu_time_us":1265,"apply.run_wall_time_us":1266,"num_ops":1,"prepare.queue_time_us":4727,"prepare.run_cpu_time_us":1794,"prepare.run_wall_time_us":1794,"raft.queue_time_us":932,"raft.run_cpu_time_us":588,"raft.run_wall_time_us":588,"replication_time_us":19554}]]}
I20250114 02:54:34.607887 2143 raft_consensus.cc:3054] T f40c66314342453b898da3ac0e7e86bf P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:34.608450 471 master_service.cc:946] ReplaceTablet: received request to replace tablet e285f86c29654b9697427516e99ca43a from {username='slave'} at 127.0.0.1:33540
W20250114 02:54:34.522413 466 rpcz_store.cc:269] 0114 02:54:30.341080 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:30.586055 (+244975us) service_pool.cc:225] Handling call
0114 02:54:34.521871 (+3935816us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:32.021313 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:32.021743 (+ 430us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:32.021769 (+ 26us) write_op.cc:435] Acquired schema lock
0114 02:54:32.021786 (+ 17us) tablet.cc:659] Decoding operations
0114 02:54:32.022073 (+ 287us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:32.022114 (+ 41us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:32.022145 (+ 31us) tablet.cc:682] Acquiring locks for 1 operations
0114 02:54:32.022291 (+ 146us) tablet.cc:698] Row locks acquired
0114 02:54:32.022306 (+ 15us) write_op.cc:260] PREPARE: finished
0114 02:54:32.022427 (+ 121us) write_op.cc:270] Start()
0114 02:54:32.022515 (+ 88us) write_op.cc:276] Timestamp: P: 1736823272022389 usec, L: 0
0114 02:54:32.022532 (+ 17us) op_driver.cc:348] REPLICATION: starting
0114 02:54:32.023050 (+ 518us) log.cc:844] Serialized 530 byte log entry
0114 02:54:32.030362 (+ 7312us) op_driver.cc:464] REPLICATION: finished
0114 02:54:32.030656 (+ 294us) write_op.cc:301] APPLY: starting
0114 02:54:32.030706 (+ 50us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:32.030801 (+ 95us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:32.030884 (+ 83us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:32.031069 (+ 185us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:32.031192 (+ 123us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:32.031233 (+ 41us) write_op.cc:312] APPLY: finished
0114 02:54:32.031395 (+ 162us) log.cc:844] Serialized 23 byte log entry
0114 02:54:32.031526 (+ 131us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:32.031593 (+ 67us) write_op.cc:454] Released schema lock
0114 02:54:32.031622 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:32.031824 (+ 202us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:32.031850 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":92430,"reported_tablets":33,"child_traces":[["op",{"apply.queue_time_us":159,"apply.run_cpu_time_us":1129,"apply.run_wall_time_us":1129,"num_ops":1,"prepare.queue_time_us":15259,"prepare.run_cpu_time_us":2022,"prepare.run_wall_time_us":3410,"raft.queue_time_us":79,"raft.run_cpu_time_us":598,"raft.run_wall_time_us":598,"replication_time_us":7749}]]}
I20250114 02:54:34.611857 466 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_910b0ec1594a4dfcbb342055cab3eb65" } new_table_name: "default.Table_3c0c8ba6eec34be4a6ea569d66038106" modify_external_catalogs: true
W20250114 02:54:34.612881 456 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.AlterSchema -> {remote=127.0.107.66:44037, user_credentials={real_user=slave}} blocked reactor thread for 69529.1us
I20250114 02:54:34.614324 2143 raft_consensus.cc:513] T f40c66314342453b898da3ac0e7e86bf P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:34.614607 939 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e858a8449444ef999b9bbef612249d3 with delete_type TABLET_DATA_DELETED (replaced by tablet 797d6cdf36704c98aa106aac87a945ca) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:34.616484 934 tablet_service.cc:1514] Processing DeleteTablet for tablet 3161f75bc4e84000af022faa9804c810 with delete_type TABLET_DATA_DELETED (replaced by tablet 53929328fa51408aa98d7825018b6fdb) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:34.618466 813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40c66314342453b898da3ac0e7e86bf" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:34.618978 813 raft_consensus.cc:3054] T f40c66314342453b898da3ac0e7e86bf P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
W20250114 02:54:34.618950 468 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.67:34825 (request call id 52) took 3920 ms (client timeout 1000 ms). Trace:
I20250114 02:54:34.628715 813 raft_consensus.cc:2463] T f40c66314342453b898da3ac0e7e86bf P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:34.629707 898 leader_election.cc:304] T f40c66314342453b898da3ac0e7e86bf P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
I20250114 02:54:34.576751 2420 pending_rounds.cc:62] T c86562688d5b4fffb71cbdf7be0afc37 P 525198593c8e47afaee54e74267b87e3: Trying to abort 1 pending ops.
I20250114 02:54:34.630501 2366 raft_consensus.cc:2798] T f40c66314342453b898da3ac0e7e86bf P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:34.630726 2420 pending_rounds.cc:69] T c86562688d5b4fffb71cbdf7be0afc37 P 525198593c8e47afaee54e74267b87e3: Aborting op as it isn't in flight: id { term: 2 index: 1 } timestamp: 7114028131631206400 op_type: NO_OP noop_request { }
I20250114 02:54:34.630964 2366 raft_consensus.cc:695] T f40c66314342453b898da3ac0e7e86bf P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Becoming Leader. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Running, Role: LEADER
I20250114 02:54:34.631119 2420 raft_consensus.cc:2883] T c86562688d5b4fffb71cbdf7be0afc37 P 525198593c8e47afaee54e74267b87e3 [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250114 02:54:34.631559 2420 raft_consensus.cc:2267] T c86562688d5b4fffb71cbdf7be0afc37 P 525198593c8e47afaee54e74267b87e3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:34.631709 2366 consensus_queue.cc:237] T f40c66314342453b898da3ac0e7e86bf P 525198593c8e47afaee54e74267b87e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
W20250114 02:54:34.601691 465 rpcz_store.cc:269] 0114 02:54:31.439792 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:31.455915 (+ 16123us) service_pool.cc:225] Handling call
0114 02:54:33.377037 (+1921122us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:33.377062 (+ 25us) write_op.cc:454] Released schema lock
0114 02:54:34.601248 (+1224186us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:33.307966 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:33.308344 (+ 378us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:33.308367 (+ 23us) write_op.cc:435] Acquired schema lock
0114 02:54:33.308379 (+ 12us) tablet.cc:659] Decoding operations
0114 02:54:33.308601 (+ 222us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:33.308638 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:33.308658 (+ 20us) tablet.cc:682] Acquiring locks for 1 operations
0114 02:54:33.308799 (+ 141us) tablet.cc:698] Row locks acquired
0114 02:54:33.308811 (+ 12us) write_op.cc:260] PREPARE: finished
0114 02:54:33.308917 (+ 106us) write_op.cc:270] Start()
0114 02:54:33.308990 (+ 73us) write_op.cc:276] Timestamp: P: 1736823273308859 usec, L: 0
0114 02:54:33.309004 (+ 14us) op_driver.cc:348] REPLICATION: starting
0114 02:54:33.309423 (+ 419us) log.cc:844] Serialized 530 byte log entry
0114 02:54:33.319266 (+ 9843us) op_driver.cc:464] REPLICATION: finished
0114 02:54:33.319632 (+ 366us) write_op.cc:301] APPLY: starting
0114 02:54:33.319697 (+ 65us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:33.319810 (+ 113us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:33.319914 (+ 104us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:33.320106 (+ 192us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:33.320296 (+ 190us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:33.320354 (+ 58us) write_op.cc:312] APPLY: finished
0114 02:54:33.320579 (+ 225us) log.cc:844] Serialized 23 byte log entry
0114 02:54:33.320718 (+ 139us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:33.320775 (+ 57us) write_op.cc:454] Released schema lock
0114 02:54:33.320805 (+ 30us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0114 02:54:34.386954 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:34.387288 (+ 334us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:34.387321 (+ 33us) write_op.cc:435] Acquired schema lock
0114 02:54:34.387333 (+ 12us) tablet.cc:659] Decoding operations
0114 02:54:34.387603 (+ 270us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:34.387642 (+ 39us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:34.387663 (+ 21us) tablet.cc:682] Acquiring locks for 1 operations
0114 02:54:34.387797 (+ 134us) tablet.cc:698] Row locks acquired
0114 02:54:34.387811 (+ 14us) write_op.cc:260] PREPARE: finished
0114 02:54:34.387916 (+ 105us) write_op.cc:270] Start()
0114 02:54:34.387992 (+ 76us) write_op.cc:276] Timestamp: P: 1736823274387851 usec, L: 0
0114 02:54:34.388007 (+ 15us) op_driver.cc:348] REPLICATION: starting
0114 02:54:34.388689 (+ 682us) log.cc:844] Serialized 356 byte log entry
0114 02:54:34.410993 (+ 22304us) op_driver.cc:464] REPLICATION: finished
0114 02:54:34.411630 (+ 637us) write_op.cc:301] APPLY: starting
0114 02:54:34.411688 (+ 58us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:34.411786 (+ 98us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:34.411891 (+ 105us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:34.412042 (+ 151us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:34.412187 (+ 145us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:34.412254 (+ 67us) write_op.cc:312] APPLY: finished
0114 02:54:34.412413 (+ 159us) log.cc:844] Serialized 23 byte log entry
0114 02:54:34.412545 (+ 132us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:34.412614 (+ 69us) write_op.cc:454] Released schema lock
0114 02:54:34.412642 (+ 28us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:34.412829 (+ 187us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:34.412854 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1438,"reported_tablets":36,"child_traces":[["op",{"apply.queue_time_us":191,"apply.run_cpu_time_us":1317,"apply.run_wall_time_us":1317,"num_ops":1,"prepare.queue_time_us":92,"prepare.run_cpu_time_us":1707,"prepare.run_wall_time_us":1707,"raft.queue_time_us":155,"raft.run_cpu_time_us":561,"raft.run_wall_time_us":561,"replication_time_us":10134}],["op",{"apply.queue_time_us":483,"apply.run_cpu_time_us":1175,"apply.run_wall_time_us":1176,"num_ops":1,"prepare.queue_time_us":23186,"prepare.run_cpu_time_us":1877,"prepare.run_wall_time_us":7862,"raft.queue_time_us":496,"raft.run_cpu_time_us":533,"raft.run_wall_time_us":533,"replication_time_us":22902}]]}
I20250114 02:54:34.635268 465 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_fd43493446ca49a48e8a970bb92ea239" } modify_external_catalogs: true
I20250114 02:54:34.635811 465 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_fd43493446ca49a48e8a970bb92ea239" } modify_external_catalogs: true
I20250114 02:54:34.636235 1992 ts_tablet_manager.cc:1918] T 99b940d872ce4781907517c40fe08781 P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250114 02:54:34.636611 1992 log.cc:1198] T 99b940d872ce4781907517c40fe08781 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/99b940d872ce4781907517c40fe08781
I20250114 02:54:34.638469 1099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40c66314342453b898da3ac0e7e86bf" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
W20250114 02:54:34.639696 898 leader_election.cc:343] T f40c66314342453b898da3ac0e7e86bf P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
W20250114 02:54:34.640425 469 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.66:50663 (request call id 54) took 3080 ms (client timeout 1000 ms). Trace:
I20250114 02:54:34.616015 2143 leader_election.cc:290] T f40c66314342453b898da3ac0e7e86bf P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Requested vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:34.649556 1992 ts_tablet_manager.cc:1939] T 99b940d872ce4781907517c40fe08781 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
W20250114 02:54:34.503363 464 rpcz_store.cc:269] 0114 02:54:30.422101 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:30.589670 (+167569us) service_pool.cc:225] Handling call
0114 02:54:30.773888 (+184218us) spinlock_profiling.cc:243] Waited 118 ms on lock 0x7b54000316d4. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616cad50 00007f646174df2a 00007f64633f2dec 00007f64634eadee 00007f6461248fc4 00007f6461248f52 00007f6461248e81 00007f6461248dfc 00007f64612480b2 00007f645f512dfc 00007f645f512236
0114 02:54:33.103690 (+2329802us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:33.103712 (+ 22us) write_op.cc:454] Released schema lock
0114 02:54:34.503032 (+1399320us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:33.037119 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:33.037543 (+ 424us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:33.037579 (+ 36us) write_op.cc:435] Acquired schema lock
0114 02:54:33.037596 (+ 17us) tablet.cc:659] Decoding operations
0114 02:54:33.038054 (+ 458us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:33.038100 (+ 46us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:33.038132 (+ 32us) tablet.cc:682] Acquiring locks for 3 operations
0114 02:54:33.038397 (+ 265us) tablet.cc:698] Row locks acquired
0114 02:54:33.038413 (+ 16us) write_op.cc:260] PREPARE: finished
0114 02:54:33.038565 (+ 152us) write_op.cc:270] Start()
0114 02:54:33.038662 (+ 97us) write_op.cc:276] Timestamp: P: 1736823273038499 usec, L: 0
0114 02:54:33.038681 (+ 19us) op_driver.cc:348] REPLICATION: starting
0114 02:54:33.039210 (+ 529us) log.cc:844] Serialized 1284 byte log entry
0114 02:54:33.045447 (+ 6237us) op_driver.cc:464] REPLICATION: finished
0114 02:54:33.048358 (+ 2911us) write_op.cc:301] APPLY: starting
0114 02:54:33.048416 (+ 58us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:33.049136 (+ 720us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:33.049221 (+ 85us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:33.049553 (+ 332us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:33.049687 (+ 134us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0114 02:54:33.049729 (+ 42us) write_op.cc:312] APPLY: finished
0114 02:54:33.049913 (+ 184us) log.cc:844] Serialized 35 byte log entry
0114 02:54:33.050049 (+ 136us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:33.050119 (+ 70us) write_op.cc:454] Released schema lock
0114 02:54:33.050157 (+ 38us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:33.050343 (+ 186us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:33.050364 (+ 21us) write_op.cc:454] Released schema lock
Related trace 'op':
0114 02:54:33.088760 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:33.089293 (+ 533us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:33.089331 (+ 38us) write_op.cc:435] Acquired schema lock
0114 02:54:33.089350 (+ 19us) tablet.cc:659] Decoding operations
0114 02:54:33.089705 (+ 355us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:33.089757 (+ 52us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:33.089790 (+ 33us) tablet.cc:682] Acquiring locks for 1 operations
0114 02:54:33.089960 (+ 170us) tablet.cc:698] Row locks acquired
0114 02:54:33.089978 (+ 18us) write_op.cc:260] PREPARE: finished
0114 02:54:33.090106 (+ 128us) write_op.cc:270] Start()
0114 02:54:33.090205 (+ 99us) write_op.cc:276] Timestamp: P: 1736823273090060 usec, L: 0
0114 02:54:33.090226 (+ 21us) op_driver.cc:348] REPLICATION: starting
0114 02:54:33.090807 (+ 581us) log.cc:844] Serialized 356 byte log entry
0114 02:54:33.098410 (+ 7603us) op_driver.cc:464] REPLICATION: finished
0114 02:54:33.102528 (+ 4118us) write_op.cc:301] APPLY: starting
0114 02:54:33.102579 (+ 51us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:33.102668 (+ 89us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:33.102755 (+ 87us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:33.102906 (+ 151us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:33.103027 (+ 121us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:33.103068 (+ 41us) write_op.cc:312] APPLY: finished
0114 02:54:33.103222 (+ 154us) log.cc:844] Serialized 23 byte log entry
0114 02:54:33.103356 (+ 134us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:33.103414 (+ 58us) write_op.cc:454] Released schema lock
0114 02:54:33.103442 (+ 28us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":91125,"reported_tablets":30,"spinlock_wait_cycles":270426880,"child_traces":[["op",{"apply.queue_time_us":2768,"apply.run_cpu_time_us":1388,"apply.run_wall_time_us":1963,"num_ops":3,"prepare.queue_time_us":98,"prepare.run_cpu_time_us":2383,"prepare.run_wall_time_us":2383,"raft.queue_time_us":92,"raft.run_cpu_time_us":608,"raft.run_wall_time_us":608,"replication_time_us":6687}],["op",{"apply.queue_time_us":3982,"apply.run_cpu_time_us":1070,"apply.run_wall_time_us":1069,"num_ops":1,"prepare.queue_time_us":11288,"prepare.run_cpu_time_us":2388,"prepare.run_wall_time_us":2388,"raft.queue_time_us":191,"raft.run_cpu_time_us":744,"raft.run_wall_time_us":745,"replication_time_us":8091}]]}
I20250114 02:54:34.654279 2247 consensus_queue.cc:1035] T 967f4d5b944543d49e87645d069b5b55 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:54:34.665937 1207 tablet_bootstrap.cc:654] T d2a7b2bee5c1440fb705b172c18a9247 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:34.670512 1206 tablet_bootstrap.cc:492] T d56255ca806d4e2cac09fbb5e1b1efbd P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:34.670961 1206 ts_tablet_manager.cc:1397] T d56255ca806d4e2cac09fbb5e1b1efbd P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.227s user 0.008s sys 0.000s
W20250114 02:54:34.662663 466 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d2a7b2bee5c1440fb705b172c18a9247 consensus state has no leader
I20250114 02:54:34.697378 466 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:33540 (request call id 319) took 3208 ms. Trace:
I20250114 02:54:34.697662 1205 tablet_bootstrap.cc:654] T 0b8989d340fa4ae992ef97e226d41110 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
W20250114 02:54:34.640738 469 rpcz_store.cc:269] 0114 02:54:31.559799 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:31.633198 (+ 73399us) service_pool.cc:225] Handling call
0114 02:54:31.692388 (+ 59190us) spinlock_profiling.cc:243] Waited 48.2 ms on lock 0x7b54000316d4. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616cad50 00007f646174df2a 00007f64633f2dec 00007f64634eadee 00007f6461248fc4 00007f6461248f52 00007f6461248e81 00007f6461248dfc 00007f64612480b2 00007f645f512dfc 00007f645f512236
0114 02:54:34.640365 (+2947977us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:33.593272 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:33.593692 (+ 420us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:33.593722 (+ 30us) write_op.cc:435] Acquired schema lock
0114 02:54:33.593738 (+ 16us) tablet.cc:659] Decoding operations
0114 02:54:33.594086 (+ 348us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:33.594133 (+ 47us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:33.594159 (+ 26us) tablet.cc:682] Acquiring locks for 2 operations
0114 02:54:33.594490 (+ 331us) tablet.cc:698] Row locks acquired
0114 02:54:33.594507 (+ 17us) write_op.cc:260] PREPARE: finished
0114 02:54:33.594636 (+ 129us) write_op.cc:270] Start()
0114 02:54:33.594726 (+ 90us) write_op.cc:276] Timestamp: P: 1736823273594584 usec, L: 0
0114 02:54:33.594743 (+ 17us) op_driver.cc:348] REPLICATION: starting
0114 02:54:33.595272 (+ 529us) log.cc:844] Serialized 907 byte log entry
0114 02:54:33.603774 (+ 8502us) op_driver.cc:464] REPLICATION: finished
0114 02:54:33.604013 (+ 239us) write_op.cc:301] APPLY: starting
0114 02:54:33.604067 (+ 54us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:33.604167 (+ 100us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:33.604277 (+ 110us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:33.604532 (+ 255us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:33.604676 (+ 144us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0114 02:54:33.604719 (+ 43us) write_op.cc:312] APPLY: finished
0114 02:54:33.605440 (+ 721us) log.cc:844] Serialized 29 byte log entry
0114 02:54:33.605578 (+ 138us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:33.605645 (+ 67us) write_op.cc:454] Released schema lock
0114 02:54:33.605674 (+ 29us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":51232,"reported_tablets":35,"spinlock_wait_cycles":110771968,"child_traces":[["op",{"apply.queue_time_us":110,"apply.run_cpu_time_us":1848,"apply.run_wall_time_us":1848,"num_ops":2,"prepare.queue_time_us":100,"prepare.run_cpu_time_us":2272,"prepare.run_wall_time_us":2273,"raft.queue_time_us":996,"raft.run_cpu_time_us":528,"raft.run_wall_time_us":528,"replication_time_us":8950}]]}
I20250114 02:54:34.697685 466 rpcz_store.cc:276] 0114 02:54:31.488963 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:34.611561 (+3122598us) service_pool.cc:225] Handling call
0114 02:54:34.612411 (+ 850us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0114 02:54:34.613897 (+ 1486us) catalog_manager.cc:3627] Apply alter schema
0114 02:54:34.615104 (+ 1207us) catalog_manager.cc:3659] Acquired catalog manager lock
0114 02:54:34.615980 (+ 876us) catalog_manager.cc:3817] Updating metadata on disk
0114 02:54:34.662083 (+ 46103us) catalog_manager.cc:3848] Committing alterations to in-memory state
0114 02:54:34.697320 (+ 35237us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:34.618866 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:34.619210 (+ 344us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:34.619234 (+ 24us) write_op.cc:435] Acquired schema lock
0114 02:54:34.619247 (+ 13us) tablet.cc:659] Decoding operations
0114 02:54:34.619493 (+ 246us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:34.619530 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:34.619554 (+ 24us) tablet.cc:682] Acquiring locks for 1 operations
0114 02:54:34.619691 (+ 137us) tablet.cc:698] Row locks acquired
0114 02:54:34.619704 (+ 13us) write_op.cc:260] PREPARE: finished
0114 02:54:34.619809 (+ 105us) write_op.cc:270] Start()
0114 02:54:34.619880 (+ 71us) write_op.cc:276] Timestamp: P: 1736823274619775 usec, L: 0
0114 02:54:34.619894 (+ 14us) op_driver.cc:348] REPLICATION: starting
0114 02:54:34.620349 (+ 455us) log.cc:844] Serialized 405 byte log entry
0114 02:54:34.637983 (+ 17634us) op_driver.cc:464] REPLICATION: finished
0114 02:54:34.638699 (+ 716us) write_op.cc:301] APPLY: starting
0114 02:54:34.638756 (+ 57us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:34.638871 (+ 115us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:34.638960 (+ 89us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:34.639107 (+ 147us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:34.639240 (+ 133us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:34.639283 (+ 43us) write_op.cc:312] APPLY: finished
0114 02:54:34.639466 (+ 183us) log.cc:844] Serialized 23 byte log entry
0114 02:54:34.639606 (+ 140us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:34.639661 (+ 55us) write_op.cc:454] Released schema lock
0114 02:54:34.639689 (+ 28us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:34.639877 (+ 188us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:34.639903 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":559,"apply.run_cpu_time_us":1157,"apply.run_wall_time_us":1157,"num_ops":1,"prepare.queue_time_us":1792,"prepare.run_cpu_time_us":1726,"prepare.run_wall_time_us":1726,"raft.queue_time_us":3063,"raft.run_cpu_time_us":701,"raft.run_wall_time_us":701,"replication_time_us":18012}]]}
I20250114 02:54:34.666934 471 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet e285f86c29654b9697427516e99ca43a
I20250114 02:54:34.702227 471 catalog_manager.cc:6596] ReplaceTablet: tablet e285f86c29654b9697427516e99ca43a deleted and replaced by tablet fd1aa3d4c6fb4bf2b1265c96b3177629
I20250114 02:54:34.702570 471 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:33540 (request call id 318) took 3218 ms. Trace:
I20250114 02:54:34.703259 1992 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 294) took 3783 ms. Trace:
I20250114 02:54:34.703583 1992 rpcz_store.cc:276] 0114 02:54:30.919644 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:30.919775 (+ 131us) service_pool.cc:225] Handling call
0114 02:54:34.020382 (+3100607us) ts_tablet_manager.cc:1145] Deleting tablet 99b940d872ce4781907517c40fe08781
0114 02:54:34.020431 (+ 49us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:34.275270 (+254839us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:34.635999 (+360729us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:34.703202 (+ 67203us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3100050}
W20250114 02:54:34.680999 457 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet d2a7b2bee5c1440fb705b172c18a9247 (table default.Table_3c0c8ba6eec34be4a6ea569d66038106 [id=4d2f7fabf8a44155a586a2b79a8fca53], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet d2a7b2bee5c1440fb705b172c18a9247 consensus state has no leader
I20250114 02:54:34.705258 1992 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 295) took 3666 ms. Trace:
I20250114 02:54:34.705535 1992 rpcz_store.cc:276] 0114 02:54:31.038394 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:31.038507 (+ 113us) service_pool.cc:225] Handling call
0114 02:54:34.704843 (+3666336us) ts_tablet_manager.cc:1145] Deleting tablet 41814f82964a4bcb8dbb6d3738052448
0114 02:54:34.704901 (+ 58us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:34.705207 (+ 306us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3665793}
I20250114 02:54:34.706522 934 tablet_service.cc:1514] Processing DeleteTablet for tablet e285f86c29654b9697427516e99ca43a with delete_type TABLET_DATA_DELETED (replaced by tablet fd1aa3d4c6fb4bf2b1265c96b3177629) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:34.707000 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet 99b940d872ce4781907517c40fe08781 (table default.Table_62251807af5845a89e89f8b2e003928f [id=e3418787ef0a4e22979fb41d4f94ebb0]) successfully deleted
W20250114 02:54:34.707098 2393 log.cc:926] Time spent T 967f4d5b944543d49e87645d069b5b55 P 525198593c8e47afaee54e74267b87e3: Append to log took a long time: real 0.057s user 0.000s sys 0.010s
W20250114 02:54:34.707616 458 catalog_manager.cc:4852] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet 41814f82964a4bcb8dbb6d3738052448 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 41814f82964a4bcb8dbb6d3738052448 already in progress: creating tablet
I20250114 02:54:34.707722 2420 ts_tablet_manager.cc:1905] T c86562688d5b4fffb71cbdf7be0afc37 P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:34.708738 1075 tablet_service.cc:1514] Processing DeleteTablet for tablet e285f86c29654b9697427516e99ca43a with delete_type TABLET_DATA_DELETED (replaced by tablet fd1aa3d4c6fb4bf2b1265c96b3177629) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:34.709924 797 tablet_service.cc:1514] Processing DeleteTablet for tablet e285f86c29654b9697427516e99ca43a with delete_type TABLET_DATA_DELETED (replaced by tablet fd1aa3d4c6fb4bf2b1265c96b3177629) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:34.702849 471 rpcz_store.cc:276] 0114 02:54:31.484340 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:34.608326 (+3123986us) service_pool.cc:225] Handling call
0114 02:54:34.702493 (+ 94167us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:34.621753 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:34.622056 (+ 303us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:34.622077 (+ 21us) write_op.cc:435] Acquired schema lock
0114 02:54:34.622089 (+ 12us) tablet.cc:659] Decoding operations
0114 02:54:34.622373 (+ 284us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:34.622405 (+ 32us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:34.622426 (+ 21us) tablet.cc:682] Acquiring locks for 2 operations
0114 02:54:34.622597 (+ 171us) tablet.cc:698] Row locks acquired
0114 02:54:34.622610 (+ 13us) write_op.cc:260] PREPARE: finished
0114 02:54:34.622704 (+ 94us) write_op.cc:270] Start()
0114 02:54:34.622776 (+ 72us) write_op.cc:276] Timestamp: P: 1736823274622671 usec, L: 0
0114 02:54:34.622790 (+ 14us) op_driver.cc:348] REPLICATION: starting
0114 02:54:34.623224 (+ 434us) log.cc:844] Serialized 625 byte log entry
0114 02:54:34.638182 (+ 14958us) op_driver.cc:464] REPLICATION: finished
0114 02:54:34.640129 (+ 1947us) write_op.cc:301] APPLY: starting
0114 02:54:34.640170 (+ 41us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:34.640287 (+ 117us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:34.640378 (+ 91us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:34.640678 (+ 300us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:34.640903 (+ 225us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:34.640942 (+ 39us) write_op.cc:312] APPLY: finished
0114 02:54:34.641098 (+ 156us) log.cc:844] Serialized 29 byte log entry
0114 02:54:34.641227 (+ 129us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:34.641288 (+ 61us) write_op.cc:454] Released schema lock
0114 02:54:34.641316 (+ 28us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1844,"apply.run_cpu_time_us":1339,"apply.run_wall_time_us":1339,"mutex_wait_us":2864,"num_ops":2,"prepare.queue_time_us":2956,"prepare.run_cpu_time_us":1737,"prepare.run_wall_time_us":1737,"raft.queue_time_us":997,"raft.run_cpu_time_us":10,"raft.run_wall_time_us":10,"replication_time_us":15331,"thread_start_us":34931,"threads_started":1}]]}
W20250114 02:54:34.619256 468 rpcz_store.cc:269] 0114 02:54:30.698569 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:30.897426 (+198857us) service_pool.cc:225] Handling call
0114 02:54:33.125731 (+2228305us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:33.125753 (+ 22us) write_op.cc:454] Released schema lock
0114 02:54:34.324125 (+1198372us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:34.324148 (+ 23us) write_op.cc:454] Released schema lock
0114 02:54:34.618887 (+294739us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:33.111672 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:33.112009 (+ 337us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:33.112030 (+ 21us) write_op.cc:435] Acquired schema lock
0114 02:54:33.112043 (+ 13us) tablet.cc:659] Decoding operations
0114 02:54:33.112288 (+ 245us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:33.112324 (+ 36us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:33.112345 (+ 21us) tablet.cc:682] Acquiring locks for 1 operations
0114 02:54:33.112472 (+ 127us) tablet.cc:698] Row locks acquired
0114 02:54:33.112484 (+ 12us) write_op.cc:260] PREPARE: finished
0114 02:54:33.112588 (+ 104us) write_op.cc:270] Start()
0114 02:54:33.112666 (+ 78us) write_op.cc:276] Timestamp: P: 1736823273112552 usec, L: 0
0114 02:54:33.112680 (+ 14us) op_driver.cc:348] REPLICATION: starting
0114 02:54:33.113084 (+ 404us) log.cc:844] Serialized 530 byte log entry
0114 02:54:33.119691 (+ 6607us) op_driver.cc:464] REPLICATION: finished
0114 02:54:33.120044 (+ 353us) write_op.cc:301] APPLY: starting
0114 02:54:33.120092 (+ 48us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:33.120193 (+ 101us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:33.120319 (+ 126us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:33.120471 (+ 152us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:33.120595 (+ 124us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:33.120647 (+ 52us) write_op.cc:312] APPLY: finished
0114 02:54:33.120812 (+ 165us) log.cc:844] Serialized 23 byte log entry
0114 02:54:33.120945 (+ 133us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:33.121004 (+ 59us) write_op.cc:454] Released schema lock
0114 02:54:33.121032 (+ 28us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0114 02:54:34.303575 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:34.304040 (+ 465us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:34.304066 (+ 26us) write_op.cc:435] Acquired schema lock
0114 02:54:34.304079 (+ 13us) tablet.cc:659] Decoding operations
0114 02:54:34.304425 (+ 346us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:34.304465 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:34.304487 (+ 22us) tablet.cc:682] Acquiring locks for 1 operations
0114 02:54:34.304651 (+ 164us) tablet.cc:698] Row locks acquired
0114 02:54:34.304670 (+ 19us) write_op.cc:260] PREPARE: finished
0114 02:54:34.304812 (+ 142us) write_op.cc:270] Start()
0114 02:54:34.304925 (+ 113us) write_op.cc:276] Timestamp: P: 1736823274304740 usec, L: 0
0114 02:54:34.304946 (+ 21us) op_driver.cc:348] REPLICATION: starting
0114 02:54:34.305582 (+ 636us) log.cc:844] Serialized 356 byte log entry
0114 02:54:34.316358 (+ 10776us) op_driver.cc:464] REPLICATION: finished
0114 02:54:34.316720 (+ 362us) write_op.cc:301] APPLY: starting
0114 02:54:34.316790 (+ 70us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:34.316887 (+ 97us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:34.317550 (+ 663us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:34.317715 (+ 165us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:34.317843 (+ 128us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:34.317885 (+ 42us) write_op.cc:312] APPLY: finished
0114 02:54:34.318049 (+ 164us) log.cc:844] Serialized 23 byte log entry
0114 02:54:34.318179 (+ 130us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:34.318240 (+ 61us) write_op.cc:454] Released schema lock
0114 02:54:34.318267 (+ 27us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":20687,"reported_tablets":31,"child_traces":[["op",{"apply.queue_time_us":217,"apply.run_cpu_time_us":1136,"apply.run_wall_time_us":1136,"num_ops":1,"prepare.queue_time_us":89,"prepare.run_cpu_time_us":1647,"prepare.run_wall_time_us":1648,"raft.queue_time_us":163,"raft.run_cpu_time_us":615,"raft.run_wall_time_us":615,"replication_time_us":6928}],["op",{"apply.queue_time_us":195,"apply.run_cpu_time_us":1711,"apply.run_wall_time_us":1711,"num_ops":1,"prepare.queue_time_us":126,"prepare.run_cpu_time_us":2322,"prepare.run_wall_time_us":2323,"raft.queue_time_us":1394,"raft.run_cpu_time_us":735,"raft.run_wall_time_us":736,"replication_time_us":11292}]]}
I20250114 02:54:34.717746 1206 raft_consensus.cc:357] T d56255ca806d4e2cac09fbb5e1b1efbd P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:34.718420 1206 raft_consensus.cc:383] T d56255ca806d4e2cac09fbb5e1b1efbd P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:34.718683 1206 raft_consensus.cc:738] T d56255ca806d4e2cac09fbb5e1b1efbd P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:34.719296 1206 consensus_queue.cc:260] T d56255ca806d4e2cac09fbb5e1b1efbd P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:34.721809 1207 tablet_bootstrap.cc:492] T d2a7b2bee5c1440fb705b172c18a9247 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:34.722235 1207 ts_tablet_manager.cc:1397] T d2a7b2bee5c1440fb705b172c18a9247 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.225s user 0.006s sys 0.004s
I20250114 02:54:34.723382 466 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_2850a159f211445988d540f3a56ec551" } modify_external_catalogs: true
I20250114 02:54:34.723904 466 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_2850a159f211445988d540f3a56ec551" } modify_external_catalogs: true
I20250114 02:54:34.730845 464 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_c305caaf6328407e9e7e0100432f0e9d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:34.732705 471 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_4150850ec6774c4a82d174648a35d1b4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:34.733719 1207 raft_consensus.cc:357] T d2a7b2bee5c1440fb705b172c18a9247 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
W20250114 02:54:34.734236 471 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4150850ec6774c4a82d174648a35d1b4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:34.735671 468 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_4126dce6d1614249887aee151c0b1085"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:34.736454 1207 raft_consensus.cc:383] T d2a7b2bee5c1440fb705b172c18a9247 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:34.736773 1207 raft_consensus.cc:738] T d2a7b2bee5c1440fb705b172c18a9247 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:34.737445 469 master_service.cc:946] ReplaceTablet: received request to replace tablet d05089eae2b74f9fb23a226d358a59da from {username='slave'} at 127.0.0.1:33540
I20250114 02:54:34.737378 1207 consensus_queue.cc:260] T d2a7b2bee5c1440fb705b172c18a9247 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
W20250114 02:54:34.737064 468 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4126dce6d1614249887aee151c0b1085 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:34.747192 1992 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 296) took 3691 ms. Trace:
I20250114 02:54:34.752338 1992 rpcz_store.cc:276] 0114 02:54:31.056179 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:31.058952 (+ 2773us) service_pool.cc:225] Handling call
0114 02:54:34.746749 (+3687797us) ts_tablet_manager.cc:1145] Deleting tablet 587a5f39282a40d0a8c1591fdf36f341
0114 02:54:34.746806 (+ 57us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:34.747132 (+ 326us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3686845}
W20250114 02:54:34.753510 458 catalog_manager.cc:4852] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet 587a5f39282a40d0a8c1591fdf36f341 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 587a5f39282a40d0a8c1591fdf36f341 already in progress: creating tablet
I20250114 02:54:34.758955 1992 tablet_replica.cc:331] T a1cf92e5275b454ca893b12f2d90d25b P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:34.759727 1992 raft_consensus.cc:2238] T a1cf92e5275b454ca893b12f2d90d25b P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:34.760298 1992 raft_consensus.cc:2267] T a1cf92e5275b454ca893b12f2d90d25b P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:34.762319 1992 ts_tablet_manager.cc:1905] T a1cf92e5275b454ca893b12f2d90d25b P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:34.762712 1206 ts_tablet_manager.cc:1428] T d56255ca806d4e2cac09fbb5e1b1efbd P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.091s user 0.007s sys 0.000s
I20250114 02:54:34.763605 1206 tablet_bootstrap.cc:492] T 68981913851243a7a1875fc99efb3c49 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:34.763635 1207 ts_tablet_manager.cc:1428] T d2a7b2bee5c1440fb705b172c18a9247 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.041s user 0.004s sys 0.002s
I20250114 02:54:34.766047 797 tablet_service.cc:1514] Processing DeleteTablet for tablet 99b940d872ce4781907517c40fe08781 with delete_type TABLET_DATA_DELETED (replaced by tablet fbeddf984d1f466192f60033715855ad) from {username='slave'} at 127.0.0.1:53078
W20250114 02:54:34.769481 462 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.65:49805 (request call id 58) took 1953 ms (client timeout 1000 ms). Trace:
W20250114 02:54:34.769825 462 rpcz_store.cc:269] 0114 02:54:32.816285 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:33.560624 (+744339us) service_pool.cc:225] Handling call
0114 02:54:34.769417 (+1208793us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":36}
I20250114 02:54:34.770798 1207 tablet_bootstrap.cc:492] T 0b8989d340fa4ae992ef97e226d41110 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:34.774005 1205 tablet_bootstrap.cc:492] T 0b8989d340fa4ae992ef97e226d41110 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:34.774406 1205 ts_tablet_manager.cc:1397] T 0b8989d340fa4ae992ef97e226d41110 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.269s user 0.009s sys 0.000s
I20250114 02:54:34.771623 2418 ts_tablet_manager.cc:1918] T 6e858a8449444ef999b9bbef612249d3 P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:34.774842 2418 log.cc:1198] T 6e858a8449444ef999b9bbef612249d3 P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/6e858a8449444ef999b9bbef612249d3
I20250114 02:54:34.776684 2418 ts_tablet_manager.cc:1939] T 6e858a8449444ef999b9bbef612249d3 P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
I20250114 02:54:34.781118 1205 raft_consensus.cc:357] T 0b8989d340fa4ae992ef97e226d41110 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:34.781857 1205 raft_consensus.cc:383] T 0b8989d340fa4ae992ef97e226d41110 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:34.782114 1205 raft_consensus.cc:738] T 0b8989d340fa4ae992ef97e226d41110 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:34.790576 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet 6e858a8449444ef999b9bbef612249d3 (table default.Table_5a9480c6474848d6b400e7dd274d15b0 [id=a17ad30ef32f4cbcb22b5a7d1c7096ff]) successfully deleted
W20250114 02:54:34.791528 458 catalog_manager.cc:4839] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 179248cb5acf4740b268e0ec8b2852f9 because the tablet was not found. No further retry: Not found: Tablet not found: 179248cb5acf4740b268e0ec8b2852f9
I20250114 02:54:34.793732 1206 tablet_bootstrap.cc:654] T 68981913851243a7a1875fc99efb3c49 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:34.796159 1205 consensus_queue.cc:260] T 0b8989d340fa4ae992ef97e226d41110 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:34.798404 1205 ts_tablet_manager.cc:1428] T 0b8989d340fa4ae992ef97e226d41110 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.024s user 0.006s sys 0.000s
W20250114 02:54:34.799381 458 catalog_manager.cc:4839] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet de4ca83a69704a368a25954cdc1b9392 because the tablet was not found. No further retry: Not found: Tablet not found: de4ca83a69704a368a25954cdc1b9392
I20250114 02:54:34.809206 1207 tablet_bootstrap.cc:654] T 0b8989d340fa4ae992ef97e226d41110 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
W20250114 02:54:34.810259 458 catalog_manager.cc:4839] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet e15e6e52d7294aeeb4463b3a76022ec3 because the tablet was not found. No further retry: Not found: Tablet not found: e15e6e52d7294aeeb4463b3a76022ec3
W20250114 02:54:34.817030 458 catalog_manager.cc:4839] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 46360f726fad4534a2daf67581346fc1 because the tablet was not found. No further retry: Not found: Tablet not found: 46360f726fad4534a2daf67581346fc1
I20250114 02:54:34.823097 2420 ts_tablet_manager.cc:1918] T c86562688d5b4fffb71cbdf7be0afc37 P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20250114 02:54:34.823493 2420 log.cc:1198] T c86562688d5b4fffb71cbdf7be0afc37 P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/c86562688d5b4fffb71cbdf7be0afc37
W20250114 02:54:34.825012 458 catalog_manager.cc:4839] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet d3fb28717c6f4844af84bb817fe7dfc4 because the tablet was not found. No further retry: Not found: Tablet not found: d3fb28717c6f4844af84bb817fe7dfc4
W20250114 02:54:34.826448 458 catalog_manager.cc:4839] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 25e85b0c5c9a4a3ca35a001359f21167 because the tablet was not found. No further retry: Not found: Tablet not found: 25e85b0c5c9a4a3ca35a001359f21167
W20250114 02:54:34.827768 458 catalog_manager.cc:4839] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 84738a6c78b24c1fb64013048763b18f because the tablet was not found. No further retry: Not found: Tablet not found: 84738a6c78b24c1fb64013048763b18f
I20250114 02:54:34.828673 2418 tablet_replica.cc:331] T c86562688d5b4fffb71cbdf7be0afc37 P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:34.829608 2418 raft_consensus.cc:2238] T c86562688d5b4fffb71cbdf7be0afc37 P 90d6c2c58bb54e0eafb365625ac33871 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:34.830138 2418 raft_consensus.cc:2267] T c86562688d5b4fffb71cbdf7be0afc37 P 90d6c2c58bb54e0eafb365625ac33871 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:34.831957 2418 ts_tablet_manager.cc:1905] T c86562688d5b4fffb71cbdf7be0afc37 P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:34.832239 2420 ts_tablet_manager.cc:1939] T c86562688d5b4fffb71cbdf7be0afc37 P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:34.832773 1205 tablet_bootstrap.cc:492] T a83bae91c29b463aa78412ce2e152ade P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:34.846393 1206 tablet_bootstrap.cc:492] T 68981913851243a7a1875fc99efb3c49 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:34.850406 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet c86562688d5b4fffb71cbdf7be0afc37 (table default.Table_7176743e4429467481ed67254643b9f9 [id=22e634dc594d4962b404424977f2cd74]) successfully deleted
W20250114 02:54:34.851888 456 catalog_manager.cc:4839] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet 99b940d872ce4781907517c40fe08781 because the tablet was not found. No further retry: Not found: Tablet not found: 99b940d872ce4781907517c40fe08781
I20250114 02:54:34.852818 1205 tablet_bootstrap.cc:654] T a83bae91c29b463aa78412ce2e152ade P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:34.854311 2420 tablet_replica.cc:331] T ed7f575653c848d7b076c17c38c4a244 P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:34.855204 2420 raft_consensus.cc:2238] T ed7f575653c848d7b076c17c38c4a244 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:34.858237 462 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_c50c01c8bc7e44a3bd139bc573c70122"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:34.859593 462 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c50c01c8bc7e44a3bd139bc573c70122 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250114 02:54:34.864312 464 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c305caaf6328407e9e7e0100432f0e9d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:34.866008 1207 tablet_bootstrap.cc:492] T 0b8989d340fa4ae992ef97e226d41110 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:34.866454 1207 ts_tablet_manager.cc:1397] T 0b8989d340fa4ae992ef97e226d41110 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.096s user 0.008s sys 0.004s
I20250114 02:54:34.870056 1992 ts_tablet_manager.cc:1918] T a1cf92e5275b454ca893b12f2d90d25b P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20250114 02:54:34.870481 1992 log.cc:1198] T a1cf92e5275b454ca893b12f2d90d25b P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/a1cf92e5275b454ca893b12f2d90d25b
I20250114 02:54:34.871186 1992 ts_tablet_manager.cc:1939] T a1cf92e5275b454ca893b12f2d90d25b P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:34.868670 1207 raft_consensus.cc:357] T 0b8989d340fa4ae992ef97e226d41110 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:34.855640 2420 raft_consensus.cc:2267] T ed7f575653c848d7b076c17c38c4a244 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:34.871989 1207 raft_consensus.cc:383] T 0b8989d340fa4ae992ef97e226d41110 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:34.873499 2420 ts_tablet_manager.cc:1905] T ed7f575653c848d7b076c17c38c4a244 P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:34.876250 1206 ts_tablet_manager.cc:1397] T 68981913851243a7a1875fc99efb3c49 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.113s user 0.008s sys 0.006s
I20250114 02:54:34.876557 471 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33540 (request call id 323) took 2919 ms. Trace:
I20250114 02:54:34.876260 1207 raft_consensus.cc:738] T 0b8989d340fa4ae992ef97e226d41110 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:34.877388 1207 consensus_queue.cc:260] T 0b8989d340fa4ae992ef97e226d41110 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:34.878460 1206 raft_consensus.cc:357] T 68981913851243a7a1875fc99efb3c49 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:34.876856 471 rpcz_store.cc:276] 0114 02:54:31.957607 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:34.732300 (+2774693us) service_pool.cc:225] Handling call
0114 02:54:34.737285 (+ 4985us) catalog_manager.cc:2076] Acquired catalog manager lock
0114 02:54:34.738808 (+ 1523us) catalog_manager.cc:2122] Created new table and tablet info
0114 02:54:34.875938 (+137130us) catalog_manager.cc:2182] Wrote table and tablets to system table
0114 02:54:34.876261 (+ 323us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0114 02:54:34.876500 (+ 239us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:34.741439 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:34.758890 (+ 17451us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:34.758913 (+ 23us) write_op.cc:435] Acquired schema lock
0114 02:54:34.758925 (+ 12us) tablet.cc:659] Decoding operations
0114 02:54:34.759188 (+ 263us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:34.759225 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:34.759244 (+ 19us) tablet.cc:682] Acquiring locks for 2 operations
0114 02:54:34.759426 (+ 182us) tablet.cc:698] Row locks acquired
0114 02:54:34.759438 (+ 12us) write_op.cc:260] PREPARE: finished
0114 02:54:34.759538 (+ 100us) write_op.cc:270] Start()
0114 02:54:34.759612 (+ 74us) write_op.cc:276] Timestamp: P: 1736823274759492 usec, L: 0
0114 02:54:34.759626 (+ 14us) op_driver.cc:348] REPLICATION: starting
0114 02:54:34.855609 (+ 95983us) log.cc:844] Serialized 433 byte log entry
0114 02:54:34.858483 (+ 2874us) spinlock_profiling.cc:243] Waited 95.5 ms on lock 0x7b54000316d4. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616ca6d1 00007f6461741198 00007f6461bad3b1 00007f6461bacb88 00007f6461bb1821 00007f6461bb17d9 00007f6461bb1769 00007f6461bb1731 00007f6461bb095d 000000000059ea04 000000000059e839
0114 02:54:34.872568 (+ 14085us) op_driver.cc:464] REPLICATION: finished
0114 02:54:34.872983 (+ 415us) write_op.cc:301] APPLY: starting
0114 02:54:34.873033 (+ 50us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:34.873147 (+ 114us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:34.873166 (+ 19us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:34.873542 (+ 376us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:34.873674 (+ 132us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 02:54:34.873712 (+ 38us) write_op.cc:312] APPLY: finished
0114 02:54:34.873887 (+ 175us) log.cc:844] Serialized 29 byte log entry
0114 02:54:34.874022 (+ 135us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:34.874081 (+ 59us) write_op.cc:454] Released schema lock
0114 02:54:34.874112 (+ 31us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:34.874319 (+ 207us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:34.874352 (+ 33us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":256,"apply.run_cpu_time_us":1295,"apply.run_wall_time_us":1296,"num_ops":2,"prepare.queue_time_us":1455,"prepare.run_cpu_time_us":3377,"prepare.run_wall_time_us":117222,"raft.queue_time_us":3275,"raft.run_cpu_time_us":617,"raft.run_wall_time_us":617,"replication_time_us":112860,"spinlock_wait_cycles":219578112}]]}
I20250114 02:54:34.881505 1205 tablet_bootstrap.cc:492] T a83bae91c29b463aa78412ce2e152ade P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:34.881960 1205 ts_tablet_manager.cc:1397] T a83bae91c29b463aa78412ce2e152ade P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.049s user 0.008s sys 0.000s
I20250114 02:54:34.885915 1206 raft_consensus.cc:383] T 68981913851243a7a1875fc99efb3c49 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:34.886221 1206 raft_consensus.cc:738] T 68981913851243a7a1875fc99efb3c49 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:34.886806 1206 consensus_queue.cc:260] T 68981913851243a7a1875fc99efb3c49 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:34.888844 1207 ts_tablet_manager.cc:1428] T 0b8989d340fa4ae992ef97e226d41110 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.022s user 0.005s sys 0.000s
I20250114 02:54:34.889135 1206 ts_tablet_manager.cc:1428] T 68981913851243a7a1875fc99efb3c49 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20250114 02:54:34.895032 1992 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 297) took 3552 ms. Trace:
I20250114 02:54:34.895341 1992 rpcz_store.cc:276] 0114 02:54:31.342164 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:31.342283 (+ 119us) service_pool.cc:225] Handling call
0114 02:54:34.758778 (+3416495us) ts_tablet_manager.cc:1145] Deleting tablet a1cf92e5275b454ca893b12f2d90d25b
0114 02:54:34.758854 (+ 76us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:34.825001 (+ 66147us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:34.869783 (+ 44782us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:34.894976 (+ 25193us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":23424,"tablet-delete.queue_time_us":3415612}
I20250114 02:54:34.897040 1206 tablet_bootstrap.cc:492] T ed109b49f20642f988161a3df7872419 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:34.897591 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet a1cf92e5275b454ca893b12f2d90d25b (table default.Table_b146341dc9614d3d92e92e0ecfe60405 [id=9ec503e449fb4a2a96bbb77237c2fb4f]) successfully deleted
I20250114 02:54:34.884105 1205 raft_consensus.cc:357] T a83bae91c29b463aa78412ce2e152ade P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:34.900957 1205 raft_consensus.cc:383] T a83bae91c29b463aa78412ce2e152ade P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:34.901218 1205 raft_consensus.cc:738] T a83bae91c29b463aa78412ce2e152ade P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:34.901796 1205 consensus_queue.cc:260] T a83bae91c29b463aa78412ce2e152ade P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:34.903759 1205 ts_tablet_manager.cc:1428] T a83bae91c29b463aa78412ce2e152ade P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.022s user 0.006s sys 0.000s
I20250114 02:54:34.904457 1207 tablet_bootstrap.cc:492] T f40c66314342453b898da3ac0e7e86bf P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:34.905102 1992 tablet_replica.cc:331] T 2837990edc4d4bea871a8114533c9352 P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:34.905879 1992 raft_consensus.cc:2238] T 2837990edc4d4bea871a8114533c9352 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:34.906538 1992 raft_consensus.cc:2267] T 2837990edc4d4bea871a8114533c9352 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:34.908697 1992 ts_tablet_manager.cc:1905] T 2837990edc4d4bea871a8114533c9352 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:34.918798 1206 tablet_bootstrap.cc:654] T ed109b49f20642f988161a3df7872419 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:34.923084 1207 tablet_bootstrap.cc:654] T f40c66314342453b898da3ac0e7e86bf P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
W20250114 02:54:34.967272 456 catalog_manager.cc:4839] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet 84738a6c78b24c1fb64013048763b18f because the tablet was not found. No further retry: Not found: Tablet not found: 84738a6c78b24c1fb64013048763b18f
I20250114 02:54:34.971426 1992 ts_tablet_manager.cc:1918] T 2837990edc4d4bea871a8114533c9352 P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20250114 02:54:34.971793 1992 log.cc:1198] T 2837990edc4d4bea871a8114533c9352 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/2837990edc4d4bea871a8114533c9352
I20250114 02:54:34.960062 2418 ts_tablet_manager.cc:1918] T c86562688d5b4fffb71cbdf7be0afc37 P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250114 02:54:34.967959 456 catalog_manager.cc:4839] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet de4ca83a69704a368a25954cdc1b9392 because the tablet was not found. No further retry: Not found: Tablet not found: de4ca83a69704a368a25954cdc1b9392
I20250114 02:54:34.968937 468 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33540 (request call id 324) took 2808 ms. Trace:
I20250114 02:54:34.975200 1207 tablet_bootstrap.cc:492] T f40c66314342453b898da3ac0e7e86bf P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:34.976470 2418 log.cc:1198] T c86562688d5b4fffb71cbdf7be0afc37 P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/c86562688d5b4fffb71cbdf7be0afc37
I20250114 02:54:34.978036 2418 ts_tablet_manager.cc:1939] T c86562688d5b4fffb71cbdf7be0afc37 P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
I20250114 02:54:34.976202 468 rpcz_store.cc:276] 0114 02:54:32.160879 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:34.735314 (+2574435us) service_pool.cc:225] Handling call
0114 02:54:34.742861 (+ 7547us) catalog_manager.cc:2076] Acquired catalog manager lock
0114 02:54:34.744276 (+ 1415us) catalog_manager.cc:2122] Created new table and tablet info
0114 02:54:34.968345 (+224069us) catalog_manager.cc:2182] Wrote table and tablets to system table
0114 02:54:34.968650 (+ 305us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0114 02:54:34.968885 (+ 235us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:34.904128 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:34.904507 (+ 379us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:34.904528 (+ 21us) write_op.cc:435] Acquired schema lock
0114 02:54:34.904541 (+ 13us) tablet.cc:659] Decoding operations
0114 02:54:34.904791 (+ 250us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:34.904827 (+ 36us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:34.904849 (+ 22us) tablet.cc:682] Acquiring locks for 2 operations
0114 02:54:34.905049 (+ 200us) tablet.cc:698] Row locks acquired
0114 02:54:34.905062 (+ 13us) write_op.cc:260] PREPARE: finished
0114 02:54:34.905172 (+ 110us) write_op.cc:270] Start()
0114 02:54:34.905251 (+ 79us) write_op.cc:276] Timestamp: P: 1736823274905108 usec, L: 0
0114 02:54:34.905266 (+ 15us) op_driver.cc:348] REPLICATION: starting
0114 02:54:34.907818 (+ 2552us) log.cc:844] Serialized 433 byte log entry
0114 02:54:34.964609 (+ 56791us) op_driver.cc:464] REPLICATION: finished
0114 02:54:34.964875 (+ 266us) write_op.cc:301] APPLY: starting
0114 02:54:34.964938 (+ 63us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:34.965085 (+ 147us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:34.965193 (+ 108us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:34.965683 (+ 490us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:34.965867 (+ 184us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 02:54:34.965925 (+ 58us) write_op.cc:312] APPLY: finished
0114 02:54:34.966141 (+ 216us) log.cc:844] Serialized 29 byte log entry
0114 02:54:34.966302 (+ 161us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:34.966394 (+ 92us) write_op.cc:454] Released schema lock
0114 02:54:34.966435 (+ 41us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":109,"apply.run_cpu_time_us":1771,"apply.run_wall_time_us":1771,"num_ops":2,"prepare.queue_time_us":158811,"prepare.run_cpu_time_us":1850,"prepare.run_wall_time_us":4948,"replication_time_us":59232}]]}
I20250114 02:54:34.979496 468 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_81dd1d2ab8b741db9483ac53260431ef" } new_table_name: "default.Table_20609e5296574025b1567d6463933b17" modify_external_catalogs: true
W20250114 02:54:34.980465 2398 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 12631829257691315283) tablet 00000000000000000000000000000000 cur holder 0x7b58000c2100
I20250114 02:54:34.984019 1992 ts_tablet_manager.cc:1939] T 2837990edc4d4bea871a8114533c9352 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:34.970201 471 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_debb45fd7c4b4826a73d39a6a6df88c4" } new_table_name: "default.Table_7367bdf975b243aab9651828ad630606" modify_external_catalogs: true
I20250114 02:54:34.984509 469 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet d05089eae2b74f9fb23a226d358a59da
I20250114 02:54:34.984850 1992 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 298) took 3580 ms. Trace:
I20250114 02:54:34.985345 1207 ts_tablet_manager.cc:1397] T f40c66314342453b898da3ac0e7e86bf P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.081s user 0.007s sys 0.003s
I20250114 02:54:34.985146 1992 rpcz_store.cc:276] 0114 02:54:31.404205 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:31.405318 (+ 1113us) service_pool.cc:225] Handling call
0114 02:54:34.904933 (+3499615us) ts_tablet_manager.cc:1145] Deleting tablet 2837990edc4d4bea871a8114533c9352
0114 02:54:34.904984 (+ 51us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:34.945511 (+ 40527us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:34.971217 (+ 25706us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:34.984797 (+ 13580us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3498933}
I20250114 02:54:34.985715 469 catalog_manager.cc:6596] ReplaceTablet: tablet d05089eae2b74f9fb23a226d358a59da deleted and replaced by tablet a5676ce60b6b42ea94fc620be5d4ca03
I20250114 02:54:34.986014 469 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:33540 (request call id 325) took 2757 ms. Trace:
I20250114 02:54:34.987839 1207 raft_consensus.cc:357] T f40c66314342453b898da3ac0e7e86bf P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:34.988638 1207 raft_consensus.cc:383] T f40c66314342453b898da3ac0e7e86bf P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:34.988992 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet 2837990edc4d4bea871a8114533c9352 (table default.Table_ba8866c1c90041268e09ff59a46503f1 [id=0b1b2c3855f54d5c97bd2866dfbd8a5d]) successfully deleted
I20250114 02:54:34.990720 797 tablet_service.cc:1514] Processing DeleteTablet for tablet d05089eae2b74f9fb23a226d358a59da with delete_type TABLET_DATA_DELETED (replaced by tablet a5676ce60b6b42ea94fc620be5d4ca03) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:34.991797 1992 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53078 (request call id 300) took 3347 ms. Trace:
I20250114 02:54:34.990268 1075 tablet_service.cc:1514] Processing DeleteTablet for tablet d05089eae2b74f9fb23a226d358a59da with delete_type TABLET_DATA_DELETED (replaced by tablet a5676ce60b6b42ea94fc620be5d4ca03) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:34.986330 469 rpcz_store.cc:276] 0114 02:54:32.229044 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:34.737352 (+2508308us) service_pool.cc:225] Handling call
0114 02:54:34.985962 (+248610us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:34.909170 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:34.909523 (+ 353us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:34.909541 (+ 18us) write_op.cc:435] Acquired schema lock
0114 02:54:34.909553 (+ 12us) tablet.cc:659] Decoding operations
0114 02:54:34.910899 (+ 1346us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:34.910934 (+ 35us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:34.910962 (+ 28us) tablet.cc:682] Acquiring locks for 2 operations
0114 02:54:34.911377 (+ 415us) tablet.cc:698] Row locks acquired
0114 02:54:34.911390 (+ 13us) write_op.cc:260] PREPARE: finished
0114 02:54:34.911488 (+ 98us) write_op.cc:270] Start()
0114 02:54:34.911569 (+ 81us) write_op.cc:276] Timestamp: P: 1736823274911427 usec, L: 0
0114 02:54:34.911582 (+ 13us) op_driver.cc:348] REPLICATION: starting
0114 02:54:34.911982 (+ 400us) log.cc:844] Serialized 659 byte log entry
0114 02:54:34.967018 (+ 55036us) op_driver.cc:464] REPLICATION: finished
0114 02:54:34.968137 (+ 1119us) write_op.cc:301] APPLY: starting
0114 02:54:34.968232 (+ 95us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:34.968328 (+ 96us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:34.968432 (+ 104us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:34.968763 (+ 331us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:34.968948 (+ 185us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:34.968987 (+ 39us) write_op.cc:312] APPLY: finished
0114 02:54:34.979589 (+ 10602us) log.cc:844] Serialized 29 byte log entry
0114 02:54:34.979732 (+ 143us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:34.979807 (+ 75us) write_op.cc:454] Released schema lock
0114 02:54:34.979852 (+ 45us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":988,"apply.run_cpu_time_us":1583,"apply.run_wall_time_us":12183,"num_ops":2,"prepare.queue_time_us":157167,"replication_time_us":55351}]]}
I20250114 02:54:34.993875 469 rpcz_store.cc:275] Call kudu.master.MasterService.GetMasterRegistration from 127.0.0.1:52164 (request call id 0) took 1724 ms. Trace:
I20250114 02:54:34.994231 469 rpcz_store.cc:276] 0114 02:54:33.269575 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:34.993059 (+1723484us) service_pool.cc:225] Handling call
0114 02:54:34.993795 (+ 736us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:54:34.992177 1207 raft_consensus.cc:738] T f40c66314342453b898da3ac0e7e86bf P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:34.992075 1992 rpcz_store.cc:276] 0114 02:54:31.644595 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:31.645414 (+ 819us) service_pool.cc:225] Handling call
0114 02:54:34.991402 (+3345988us) ts_tablet_manager.cc:1145] Deleting tablet 232d6a697f744e179a1a8da492a7d227
0114 02:54:34.991458 (+ 56us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:34.991750 (+ 292us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3345265}
W20250114 02:54:34.997531 458 catalog_manager.cc:4852] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet 232d6a697f744e179a1a8da492a7d227 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 232d6a697f744e179a1a8da492a7d227 already in progress: creating tablet
W20250114 02:54:34.998622 458 catalog_manager.cc:4839] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet de4ca83a69704a368a25954cdc1b9392 because the tablet was not found. No further retry: Not found: Tablet not found: de4ca83a69704a368a25954cdc1b9392
W20250114 02:54:34.999243 458 catalog_manager.cc:4839] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet e15e6e52d7294aeeb4463b3a76022ec3 because the tablet was not found. No further retry: Not found: Tablet not found: e15e6e52d7294aeeb4463b3a76022ec3
W20250114 02:54:35.000886 458 catalog_manager.cc:4839] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet ed7f575653c848d7b076c17c38c4a244 because the tablet was not found. No further retry: Not found: Tablet not found: ed7f575653c848d7b076c17c38c4a244
W20250114 02:54:35.001499 458 catalog_manager.cc:4839] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet 1381e26725174b9c877075cf5444f566 because the tablet was not found. No further retry: Not found: Tablet not found: 1381e26725174b9c877075cf5444f566
W20250114 02:54:35.002089 458 catalog_manager.cc:4839] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet 6e858a8449444ef999b9bbef612249d3 because the tablet was not found. No further retry: Not found: Tablet not found: 6e858a8449444ef999b9bbef612249d3
W20250114 02:54:35.002764 458 catalog_manager.cc:4839] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet 5ac7708550954a209e2a8d217c3b39bd because the tablet was not found. No further retry: Not found: Tablet not found: 5ac7708550954a209e2a8d217c3b39bd
W20250114 02:54:35.003440 458 catalog_manager.cc:4839] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet 59a46bc865cc48df8e98598764fd1b54 because the tablet was not found. No further retry: Not found: Tablet not found: 59a46bc865cc48df8e98598764fd1b54
W20250114 02:54:35.004076 458 catalog_manager.cc:4839] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet c86562688d5b4fffb71cbdf7be0afc37 because the tablet was not found. No further retry: Not found: Tablet not found: c86562688d5b4fffb71cbdf7be0afc37
I20250114 02:54:35.005956 934 tablet_service.cc:1514] Processing DeleteTablet for tablet d05089eae2b74f9fb23a226d358a59da with delete_type TABLET_DATA_DELETED (replaced by tablet a5676ce60b6b42ea94fc620be5d4ca03) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:35.007934 467 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet df045d80d8ce4ff194a6d2039a44bce6 (table default.Table_2befe573a994473abe0556ba7e713686 [id=da0258578bb34ae29551fc593bf7d0b5]). Expected version 1 got 0
I20250114 02:54:35.009461 467 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet 141cede72b8741a8b0615d7c6bb823bb (table default.Table_4364d1df16d149d2ba10469ce7288d4e [id=13fc996d911b430290e37a94dd44b1ab]). Expected version 1 got 0
W20250114 02:54:35.009907 458 catalog_manager.cc:4839] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet d3fb28717c6f4844af84bb817fe7dfc4 because the tablet was not found. No further retry: Not found: Tablet not found: d3fb28717c6f4844af84bb817fe7dfc4
I20250114 02:54:35.011408 1992 tablet_replica.cc:331] T 97f4e061d3b94ee9982410b7ce135453 P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:35.016460 2389 raft_consensus.cc:491] T d2a7b2bee5c1440fb705b172c18a9247 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:35.016983 2389 raft_consensus.cc:513] T d2a7b2bee5c1440fb705b172c18a9247 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:35.018929 2389 leader_election.cc:290] T d2a7b2bee5c1440fb705b172c18a9247 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:35.020965 960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2a7b2bee5c1440fb705b172c18a9247" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:35.022065 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet c86562688d5b4fffb71cbdf7be0afc37 (table default.Table_7176743e4429467481ed67254643b9f9 [id=22e634dc594d4962b404424977f2cd74]) successfully deleted
I20250114 02:54:35.008774 1207 consensus_queue.cc:260] T f40c66314342453b898da3ac0e7e86bf P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:35.022262 960 raft_consensus.cc:2463] T d2a7b2bee5c1440fb705b172c18a9247 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 0.
I20250114 02:54:35.023936 1040 leader_election.cc:304] T d2a7b2bee5c1440fb705b172c18a9247 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:35.024009 813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2a7b2bee5c1440fb705b172c18a9247" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:35.024714 813 raft_consensus.cc:2463] T d2a7b2bee5c1440fb705b172c18a9247 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 0.
I20250114 02:54:35.026510 2279 raft_consensus.cc:2798] T d2a7b2bee5c1440fb705b172c18a9247 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:35.026851 2279 raft_consensus.cc:491] T d2a7b2bee5c1440fb705b172c18a9247 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:35.027169 2279 raft_consensus.cc:3054] T d2a7b2bee5c1440fb705b172c18a9247 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:35.034585 2279 raft_consensus.cc:513] T d2a7b2bee5c1440fb705b172c18a9247 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:35.037364 1207 ts_tablet_manager.cc:1428] T f40c66314342453b898da3ac0e7e86bf P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.052s user 0.005s sys 0.000s
I20250114 02:54:35.039778 1206 tablet_bootstrap.cc:492] T ed109b49f20642f988161a3df7872419 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:35.040200 1206 ts_tablet_manager.cc:1397] T ed109b49f20642f988161a3df7872419 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.143s user 0.010s sys 0.030s
I20250114 02:54:35.042680 1206 raft_consensus.cc:357] T ed109b49f20642f988161a3df7872419 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:35.043327 1206 raft_consensus.cc:383] T ed109b49f20642f988161a3df7872419 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:35.047549 960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2a7b2bee5c1440fb705b172c18a9247" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
I20250114 02:54:35.048518 1205 tablet_bootstrap.cc:492] T 5d545ac27a7f49faace0655713405cea P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:35.048619 960 raft_consensus.cc:3054] T d2a7b2bee5c1440fb705b172c18a9247 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:35.043576 1206 raft_consensus.cc:738] T ed109b49f20642f988161a3df7872419 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:35.056149 1206 consensus_queue.cc:260] T ed109b49f20642f988161a3df7872419 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:35.059518 1206 ts_tablet_manager.cc:1428] T ed109b49f20642f988161a3df7872419 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.019s user 0.005s sys 0.000s
I20250114 02:54:35.063972 2389 consensus_queue.cc:1035] T 141cede72b8741a8b0615d7c6bb823bb P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:54:35.066964 2247 raft_consensus.cc:491] T a83bae91c29b463aa78412ce2e152ade P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:35.068513 2372 raft_consensus.cc:491] T 68981913851243a7a1875fc99efb3c49 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:35.068976 2372 raft_consensus.cc:513] T 68981913851243a7a1875fc99efb3c49 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:35.069990 2279 leader_election.cc:290] T d2a7b2bee5c1440fb705b172c18a9247 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Requested vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:35.071007 2372 leader_election.cc:290] T 68981913851243a7a1875fc99efb3c49 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:35.072765 1992 raft_consensus.cc:2238] T 97f4e061d3b94ee9982410b7ce135453 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:35.073271 1992 raft_consensus.cc:2267] T 97f4e061d3b94ee9982410b7ce135453 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:35.072718 1103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68981913851243a7a1875fc99efb3c49" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:35.075019 1992 ts_tablet_manager.cc:1905] T 97f4e061d3b94ee9982410b7ce135453 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250114 02:54:35.075294 753 leader_election.cc:343] T 68981913851243a7a1875fc99efb3c49 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:35.080534 1206 tablet_bootstrap.cc:492] T e06daadd346d45969ad5efcf03bc0870 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:35.083451 469 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_fb66bdd04429412b9585afd33d5b8876"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:35.084990 469 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fb66bdd04429412b9585afd33d5b8876 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:35.090685 467 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet a52d6749ff834b62a8e20a2f45935240 (table default.Table_1c9b7896941e4289b1c12e20c210c2c1 [id=b3ee15b9f10f4d489edbde3af9034d25]). Expected version 1 got 0
I20250114 02:54:35.067497 2247 raft_consensus.cc:513] T a83bae91c29b463aa78412ce2e152ade P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:35.093619 2418 tablet_replica.cc:331] T d05089eae2b74f9fb23a226d358a59da P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:35.094131 2247 leader_election.cc:290] T a83bae91c29b463aa78412ce2e152ade P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:35.094674 2418 raft_consensus.cc:2238] T d05089eae2b74f9fb23a226d358a59da P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Raft consensus shutting down.
I20250114 02:54:35.095657 2418 raft_consensus.cc:2267] T d05089eae2b74f9fb23a226d358a59da P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250114 02:54:35.096640 458 catalog_manager.cc:4852] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet e285f86c29654b9697427516e99ca43a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e285f86c29654b9697427516e99ca43a already in progress: creating tablet
I20250114 02:54:35.096647 958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68981913851243a7a1875fc99efb3c49" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:35.097927 462 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33540 (request call id 326) took 2846 ms. Trace:
I20250114 02:54:35.099934 1207 tablet_bootstrap.cc:492] T a83bae91c29b463aa78412ce2e152ade P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:35.098215 462 rpcz_store.cc:276] 0114 02:54:32.251737 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:34.778791 (+2527054us) service_pool.cc:225] Handling call
0114 02:54:34.858065 (+ 79274us) spinlock_profiling.cc:243] Waited 73.5 ms on lock 0x7b54000316d4. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616cad50 00007f646174df2a 00007f64633f2dec 00007f64634ec306 00007f6461250044 00007f646124ffd2 00007f646124ff01 00007f646124fe7c 00007f646124f132 00007f645f512dfc 00007f645f512236
0114 02:54:34.859947 (+ 1882us) catalog_manager.cc:2076] Acquired catalog manager lock
0114 02:54:34.861934 (+ 1987us) catalog_manager.cc:2122] Created new table and tablet info
0114 02:54:35.097356 (+235422us) catalog_manager.cc:2182] Wrote table and tablets to system table
0114 02:54:35.097636 (+ 280us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0114 02:54:35.097875 (+ 239us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:35.011214 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:35.011571 (+ 357us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:35.011595 (+ 24us) write_op.cc:435] Acquired schema lock
0114 02:54:35.011607 (+ 12us) tablet.cc:659] Decoding operations
0114 02:54:35.011885 (+ 278us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:35.011935 (+ 50us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:35.011966 (+ 31us) tablet.cc:682] Acquiring locks for 2 operations
0114 02:54:35.012188 (+ 222us) tablet.cc:698] Row locks acquired
0114 02:54:35.012206 (+ 18us) write_op.cc:260] PREPARE: finished
0114 02:54:35.012370 (+ 164us) write_op.cc:270] Start()
0114 02:54:35.012465 (+ 95us) write_op.cc:276] Timestamp: P: 1736823275012291 usec, L: 0
0114 02:54:35.012485 (+ 20us) op_driver.cc:348] REPLICATION: starting
0114 02:54:35.013071 (+ 586us) log.cc:844] Serialized 433 byte log entry
0114 02:54:35.095318 (+ 82247us) op_driver.cc:464] REPLICATION: finished
0114 02:54:35.095597 (+ 279us) write_op.cc:301] APPLY: starting
0114 02:54:35.095652 (+ 55us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:35.095764 (+ 112us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:35.095858 (+ 94us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:35.096260 (+ 402us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:35.096416 (+ 156us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 02:54:35.096455 (+ 39us) write_op.cc:312] APPLY: finished
0114 02:54:35.096626 (+ 171us) log.cc:844] Serialized 29 byte log entry
0114 02:54:35.096770 (+ 144us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:35.096834 (+ 64us) write_op.cc:454] Released schema lock
0114 02:54:35.096864 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:35.097063 (+ 199us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:35.097084 (+ 21us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":168986496,"child_traces":[["op",{"apply.queue_time_us":150,"apply.run_cpu_time_us":1436,"apply.run_wall_time_us":1436,"num_ops":2,"prepare.queue_time_us":148278,"prepare.run_cpu_time_us":2183,"prepare.run_wall_time_us":27671,"raft.queue_time_us":1309,"raft.run_cpu_time_us":783,"raft.run_wall_time_us":2278,"replication_time_us":82725}]]}
I20250114 02:54:35.104516 2279 consensus_queue.cc:1035] T 59a46bc865cc48df8e98598764fd1b54 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:35.111232 2418 ts_tablet_manager.cc:1905] T d05089eae2b74f9fb23a226d358a59da P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:35.112421 467 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet edc69b66bea54625a4b15867b75986b7 (table default.Table_0a0bda55a4a9476c98d961822b8a19bd [id=fb2be5c1ca9444a197f3315ea9c59b09]). Expected version 1 got 0
I20250114 02:54:35.125389 1205 tablet_bootstrap.cc:654] T 5d545ac27a7f49faace0655713405cea P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:35.125516 464 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33540 (request call id 322) took 3278 ms. Trace:
I20250114 02:54:35.125849 464 rpcz_store.cc:276] 0114 02:54:31.846813 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:34.730428 (+2883615us) service_pool.cc:225] Handling call
0114 02:54:34.864706 (+134278us) catalog_manager.cc:2076] Acquired catalog manager lock
0114 02:54:34.866124 (+ 1418us) catalog_manager.cc:2122] Created new table and tablet info
0114 02:54:35.124903 (+258779us) catalog_manager.cc:2182] Wrote table and tablets to system table
0114 02:54:35.125210 (+ 307us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0114 02:54:35.125458 (+ 248us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:35.039010 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:35.039386 (+ 376us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:35.039410 (+ 24us) write_op.cc:435] Acquired schema lock
0114 02:54:35.039421 (+ 11us) tablet.cc:659] Decoding operations
0114 02:54:35.039677 (+ 256us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:35.039711 (+ 34us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:35.039731 (+ 20us) tablet.cc:682] Acquiring locks for 2 operations
0114 02:54:35.039917 (+ 186us) tablet.cc:698] Row locks acquired
0114 02:54:35.039931 (+ 14us) write_op.cc:260] PREPARE: finished
0114 02:54:35.086835 (+ 46904us) write_op.cc:270] Start()
0114 02:54:35.086919 (+ 84us) write_op.cc:276] Timestamp: P: 1736823275086778 usec, L: 0
0114 02:54:35.086934 (+ 15us) op_driver.cc:348] REPLICATION: starting
0114 02:54:35.087364 (+ 430us) log.cc:844] Serialized 433 byte log entry
0114 02:54:35.122546 (+ 35182us) op_driver.cc:464] REPLICATION: finished
0114 02:54:35.123489 (+ 943us) write_op.cc:301] APPLY: starting
0114 02:54:35.123542 (+ 53us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:35.123654 (+ 112us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:35.123745 (+ 91us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:35.124109 (+ 364us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:35.124306 (+ 197us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 02:54:35.124349 (+ 43us) write_op.cc:312] APPLY: finished
0114 02:54:35.124508 (+ 159us) log.cc:844] Serialized 29 byte log entry
0114 02:54:35.124637 (+ 129us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:35.124700 (+ 63us) write_op.cc:454] Released schema lock
0114 02:54:35.124729 (+ 29us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":788,"num_ops":2,"prepare.queue_time_us":171870,"prepare.run_cpu_time_us":1797,"prepare.run_wall_time_us":48562,"replication_time_us":35522}]]}
W20250114 02:54:35.131237 468 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 62af33f52a264fa8ba977b4a05e74d2b consensus state has no leader
I20250114 02:54:35.100699 811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83bae91c29b463aa78412ce2e152ade" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
W20250114 02:54:35.110327 751 leader_election.cc:343] T 68981913851243a7a1875fc99efb3c49 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:35.150660 751 leader_election.cc:304] T 68981913851243a7a1875fc99efb3c49 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:35.150976 1207 tablet_bootstrap.cc:654] T a83bae91c29b463aa78412ce2e152ade P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:35.150208 811 raft_consensus.cc:2463] T a83bae91c29b463aa78412ce2e152ade P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:35.152549 2372 raft_consensus.cc:2743] T 68981913851243a7a1875fc99efb3c49 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:35.153733 898 leader_election.cc:304] T a83bae91c29b463aa78412ce2e152ade P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
I20250114 02:54:35.154819 2447 raft_consensus.cc:2798] T a83bae91c29b463aa78412ce2e152ade P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:35.155167 2447 raft_consensus.cc:491] T a83bae91c29b463aa78412ce2e152ade P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:35.155561 2447 raft_consensus.cc:3054] T a83bae91c29b463aa78412ce2e152ade P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:35.157898 960 raft_consensus.cc:2463] T d2a7b2bee5c1440fb705b172c18a9247 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1.
I20250114 02:54:35.161031 1040 leader_election.cc:304] T d2a7b2bee5c1440fb705b172c18a9247 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters:
W20250114 02:54:35.164618 898 consensus_peers.cc:487] T 97f4e061d3b94ee9982410b7ce135453 P 525198593c8e47afaee54e74267b87e3 -> Peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Couldn't send request to peer 819f9e3412e54d44abb11d8b3ca877c2. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250114 02:54:35.167923 2406 raft_consensus.cc:2798] T d2a7b2bee5c1440fb705b172c18a9247 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:35.168401 2406 raft_consensus.cc:695] T d2a7b2bee5c1440fb705b172c18a9247 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Becoming Leader. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Running, Role: LEADER
I20250114 02:54:35.169145 2406 consensus_queue.cc:237] T d2a7b2bee5c1440fb705b172c18a9247 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:35.175540 811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2a7b2bee5c1440fb705b172c18a9247" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:35.176240 811 raft_consensus.cc:3054] T d2a7b2bee5c1440fb705b172c18a9247 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:35.180056 2447 raft_consensus.cc:513] T a83bae91c29b463aa78412ce2e152ade P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:35.185184 468 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_382cf0d973d14925b8424879a1e6d7df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:54:35.186627 468 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_382cf0d973d14925b8424879a1e6d7df in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:35.189270 811 raft_consensus.cc:2463] T d2a7b2bee5c1440fb705b172c18a9247 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1.
I20250114 02:54:35.189718 813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83bae91c29b463aa78412ce2e152ade" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:35.190330 813 raft_consensus.cc:3054] T a83bae91c29b463aa78412ce2e152ade P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:35.199642 2447 leader_election.cc:290] T a83bae91c29b463aa78412ce2e152ade P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Requested vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:35.200583 813 raft_consensus.cc:2463] T a83bae91c29b463aa78412ce2e152ade P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:35.201740 898 leader_election.cc:304] T a83bae91c29b463aa78412ce2e152ade P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
I20250114 02:54:35.202608 2447 raft_consensus.cc:2798] T a83bae91c29b463aa78412ce2e152ade P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:35.203261 2447 raft_consensus.cc:695] T a83bae91c29b463aa78412ce2e152ade P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Becoming Leader. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Running, Role: LEADER
I20250114 02:54:35.204202 2447 consensus_queue.cc:237] T a83bae91c29b463aa78412ce2e152ade P 525198593c8e47afaee54e74267b87e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:35.213997 2389 raft_consensus.cc:491] T 0b8989d340fa4ae992ef97e226d41110 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:35.214432 2389 raft_consensus.cc:513] T 0b8989d340fa4ae992ef97e226d41110 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:35.217119 2389 leader_election.cc:290] T 0b8989d340fa4ae992ef97e226d41110 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:35.217267 959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b8989d340fa4ae992ef97e226d41110" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:35.217814 959 raft_consensus.cc:2463] T 0b8989d340fa4ae992ef97e226d41110 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 0.
I20250114 02:54:35.218765 1040 leader_election.cc:304] T 0b8989d340fa4ae992ef97e226d41110 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:35.219377 2389 raft_consensus.cc:2798] T 0b8989d340fa4ae992ef97e226d41110 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:35.219693 2389 raft_consensus.cc:491] T 0b8989d340fa4ae992ef97e226d41110 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:35.219987 2389 raft_consensus.cc:3054] T 0b8989d340fa4ae992ef97e226d41110 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:35.230337 1205 tablet_bootstrap.cc:492] T 5d545ac27a7f49faace0655713405cea P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:35.235589 1103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83bae91c29b463aa78412ce2e152ade" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:35.238107 467 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet 38fe8829d98547fb907ef78099df3d7f (table default.Table_60450873174a4c80aa7e2122343d82be [id=9299683349354f879847b7c9d93f6072]). Expected version 1 got 0
I20250114 02:54:35.238852 1205 ts_tablet_manager.cc:1397] T 5d545ac27a7f49faace0655713405cea P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.191s user 0.012s sys 0.010s
I20250114 02:54:35.238970 467 catalog_manager.cc:5526] T fc406eab968745d893f1a8538ff61f37 P 90d6c2c58bb54e0eafb365625ac33871 reported cstate change: term changed from 1 to 2, leader changed from 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67) to 525198593c8e47afaee54e74267b87e3 (127.0.107.66). New cstate: current_term: 2 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } }
I20250114 02:54:35.245127 1207 tablet_bootstrap.cc:492] T a83bae91c29b463aa78412ce2e152ade P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:35.245558 1206 tablet_bootstrap.cc:654] T e06daadd346d45969ad5efcf03bc0870 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:35.246030 1207 ts_tablet_manager.cc:1397] T a83bae91c29b463aa78412ce2e152ade P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.146s user 0.011s sys 0.000s
I20250114 02:54:35.257326 1103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83bae91c29b463aa78412ce2e152ade" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:35.269032 2389 raft_consensus.cc:513] T 0b8989d340fa4ae992ef97e226d41110 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:35.271346 2389 leader_election.cc:290] T 0b8989d340fa4ae992ef97e226d41110 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Requested vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:35.272166 960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b8989d340fa4ae992ef97e226d41110" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
I20250114 02:54:35.272856 960 raft_consensus.cc:3054] T 0b8989d340fa4ae992ef97e226d41110 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:35.273627 1992 ts_tablet_manager.cc:1918] T 97f4e061d3b94ee9982410b7ce135453 P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:35.273967 1992 log.cc:1198] T 97f4e061d3b94ee9982410b7ce135453 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/97f4e061d3b94ee9982410b7ce135453
I20250114 02:54:35.274660 1992 ts_tablet_manager.cc:1939] T 97f4e061d3b94ee9982410b7ce135453 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:35.280125 811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b8989d340fa4ae992ef97e226d41110" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:35.280752 811 raft_consensus.cc:2463] T 0b8989d340fa4ae992ef97e226d41110 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 0.
I20250114 02:54:35.283594 960 raft_consensus.cc:2463] T 0b8989d340fa4ae992ef97e226d41110 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1.
I20250114 02:54:35.284857 1040 leader_election.cc:304] T 0b8989d340fa4ae992ef97e226d41110 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:35.286448 467 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet 719188f639724851a5fc2b37cb60fdbb (table default.Table_8f8a18d7868c44bbb4d66a0692b08186 [id=e6292728826f437cb75c524a8aef2e87]). Expected version 1 got 0
W20250114 02:54:35.288419 898 leader_election.cc:343] T a83bae91c29b463aa78412ce2e152ade P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:35.290059 1207 raft_consensus.cc:357] T a83bae91c29b463aa78412ce2e152ade P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:35.290733 1207 raft_consensus.cc:383] T a83bae91c29b463aa78412ce2e152ade P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:35.290972 1207 raft_consensus.cc:738] T a83bae91c29b463aa78412ce2e152ade P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:35.291419 2389 raft_consensus.cc:2798] T 0b8989d340fa4ae992ef97e226d41110 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:35.293521 1205 raft_consensus.cc:357] T 5d545ac27a7f49faace0655713405cea P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:35.294200 1205 raft_consensus.cc:383] T 5d545ac27a7f49faace0655713405cea P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:35.294421 1205 raft_consensus.cc:738] T 5d545ac27a7f49faace0655713405cea P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:35.295648 1205 consensus_queue.cc:260] T 5d545ac27a7f49faace0655713405cea P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:35.297422 2420 ts_tablet_manager.cc:1918] T ed7f575653c848d7b076c17c38c4a244 P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:35.297869 2420 log.cc:1198] T ed7f575653c848d7b076c17c38c4a244 P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/ed7f575653c848d7b076c17c38c4a244
I20250114 02:54:35.298733 2420 ts_tablet_manager.cc:1939] T ed7f575653c848d7b076c17c38c4a244 P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
W20250114 02:54:35.301285 898 leader_election.cc:343] T a83bae91c29b463aa78412ce2e152ade P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:35.307333 2389 raft_consensus.cc:695] T 0b8989d340fa4ae992ef97e226d41110 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Becoming Leader. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Running, Role: LEADER
I20250114 02:54:35.308257 2389 consensus_queue.cc:237] T 0b8989d340fa4ae992ef97e226d41110 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:35.329564 1205 ts_tablet_manager.cc:1428] T 5d545ac27a7f49faace0655713405cea P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.090s user 0.005s sys 0.000s
I20250114 02:54:35.334281 1205 tablet_bootstrap.cc:492] T 68981913851243a7a1875fc99efb3c49 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:35.334847 2406 consensus_queue.cc:1035] T 141cede72b8741a8b0615d7c6bb823bb P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250114 02:54:35.336412 1042 consensus_peers.cc:487] T 59a46bc865cc48df8e98598764fd1b54 P 90d6c2c58bb54e0eafb365625ac33871 -> Peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Couldn't send request to peer 819f9e3412e54d44abb11d8b3ca877c2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 59a46bc865cc48df8e98598764fd1b54. This is attempt 1: this message will repeat every 5th retry.
I20250114 02:54:35.291816 1207 consensus_queue.cc:260] T a83bae91c29b463aa78412ce2e152ade P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:35.337021 2418 ts_tablet_manager.cc:1918] T d05089eae2b74f9fb23a226d358a59da P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:35.337395 2418 log.cc:1198] T d05089eae2b74f9fb23a226d358a59da P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/d05089eae2b74f9fb23a226d358a59da
I20250114 02:54:35.338171 2418 ts_tablet_manager.cc:1939] T d05089eae2b74f9fb23a226d358a59da P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
I20250114 02:54:35.339200 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet ed7f575653c848d7b076c17c38c4a244 (table default.Table_f842a61f536940588cd46bedc206f2b3 [id=954669573a3c4a97a488805e05b0b220]) successfully deleted
W20250114 02:54:35.341989 456 catalog_manager.cc:4839] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet e15e6e52d7294aeeb4463b3a76022ec3 because the tablet was not found. No further retry: Not found: Tablet not found: e15e6e52d7294aeeb4463b3a76022ec3
I20250114 02:54:35.343030 1207 ts_tablet_manager.cc:1428] T a83bae91c29b463aa78412ce2e152ade P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.096s user 0.006s sys 0.000s
I20250114 02:54:35.344969 811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b8989d340fa4ae992ef97e226d41110" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:35.345515 811 raft_consensus.cc:3054] T 0b8989d340fa4ae992ef97e226d41110 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:35.352010 811 raft_consensus.cc:2463] T 0b8989d340fa4ae992ef97e226d41110 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1.
W20250114 02:54:35.358367 456 catalog_manager.cc:4839] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet 25e85b0c5c9a4a3ca35a001359f21167 because the tablet was not found. No further retry: Not found: Tablet not found: 25e85b0c5c9a4a3ca35a001359f21167
I20250114 02:54:35.360136 2406 raft_consensus.cc:491] T a83bae91c29b463aa78412ce2e152ade P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:35.362571 2406 raft_consensus.cc:513] T a83bae91c29b463aa78412ce2e152ade P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:35.365993 2406 leader_election.cc:290] T a83bae91c29b463aa78412ce2e152ade P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:35.366751 960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83bae91c29b463aa78412ce2e152ade" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:35.368777 1207 tablet_bootstrap.cc:492] T 0b8361cd538046e0bcb4aac821748f67 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:35.370359 2420 tablet_replica.cc:331] T 6e858a8449444ef999b9bbef612249d3 P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:35.373221 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet 97f4e061d3b94ee9982410b7ce135453 (table default.Table_6a4fe39298514187b65def5841d3a992 [id=5c5a7974ad2f4e58ba5c47a6eeaaac62]) successfully deleted
W20250114 02:54:35.374723 458 catalog_manager.cc:4839] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet 3161f75bc4e84000af022faa9804c810 because the tablet was not found. No further retry: Not found: Tablet not found: 3161f75bc4e84000af022faa9804c810
W20250114 02:54:35.376538 458 catalog_manager.cc:4852] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet e285f86c29654b9697427516e99ca43a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e285f86c29654b9697427516e99ca43a already in progress: creating tablet
W20250114 02:54:35.377717 458 catalog_manager.cc:4839] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet 99b940d872ce4781907517c40fe08781 because the tablet was not found. No further retry: Not found: Tablet not found: 99b940d872ce4781907517c40fe08781
I20250114 02:54:35.378439 1992 tablet_replica.cc:331] T d05089eae2b74f9fb23a226d358a59da P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:35.393296 2420 raft_consensus.cc:2238] T 6e858a8449444ef999b9bbef612249d3 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:35.397444 2420 raft_consensus.cc:2267] T 6e858a8449444ef999b9bbef612249d3 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:35.401362 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet d05089eae2b74f9fb23a226d358a59da (table default.Table_5e1c3ceefda2466da137a659035f9ae4 [id=ee635d0ed76b46c9b2e139a75296017b]) successfully deleted
I20250114 02:54:35.405419 468 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_5940a20d29474d1a8dcc1d01e3599699"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:35.394930 1205 tablet_bootstrap.cc:654] T 68981913851243a7a1875fc99efb3c49 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:35.409147 2420 ts_tablet_manager.cc:1905] T 6e858a8449444ef999b9bbef612249d3 P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250114 02:54:35.421084 468 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5940a20d29474d1a8dcc1d01e3599699 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:35.429935 1205 tablet_bootstrap.cc:492] T 68981913851243a7a1875fc99efb3c49 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:35.430352 1205 ts_tablet_manager.cc:1397] T 68981913851243a7a1875fc99efb3c49 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.096s user 0.006s sys 0.002s
I20250114 02:54:35.436578 1207 tablet_bootstrap.cc:654] T 0b8361cd538046e0bcb4aac821748f67 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:35.436535 429 master-stress-test.cc:577] Killing test thread 1/10
I20250114 02:54:35.438563 1205 raft_consensus.cc:357] T 68981913851243a7a1875fc99efb3c49 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:35.439229 1205 raft_consensus.cc:383] T 68981913851243a7a1875fc99efb3c49 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:35.439471 1205 raft_consensus.cc:738] T 68981913851243a7a1875fc99efb3c49 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:35.440093 1205 consensus_queue.cc:260] T 68981913851243a7a1875fc99efb3c49 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:35.442104 429 master-stress-test.cc:579] Killed test thread 1/10
I20250114 02:54:35.442373 429 master-stress-test.cc:577] Killing test thread 2/10
I20250114 02:54:35.442608 429 master-stress-test.cc:579] Killed test thread 2/10
I20250114 02:54:35.442811 429 master-stress-test.cc:577] Killing test thread 3/10
I20250114 02:54:35.386785 813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83bae91c29b463aa78412ce2e152ade" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:35.449244 813 raft_consensus.cc:2388] T a83bae91c29b463aa78412ce2e152ade P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 90d6c2c58bb54e0eafb365625ac33871 in current term 1: Already voted for candidate 525198593c8e47afaee54e74267b87e3 in this term.
I20250114 02:54:35.450253 1042 leader_election.cc:304] T a83bae91c29b463aa78412ce2e152ade P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90d6c2c58bb54e0eafb365625ac33871; no voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2
I20250114 02:54:35.450893 2406 raft_consensus.cc:3054] T a83bae91c29b463aa78412ce2e152ade P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:35.413901 1206 tablet_bootstrap.cc:492] T e06daadd346d45969ad5efcf03bc0870 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:35.456707 1206 ts_tablet_manager.cc:1397] T e06daadd346d45969ad5efcf03bc0870 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.376s user 0.006s sys 0.004s
I20250114 02:54:35.457921 1205 ts_tablet_manager.cc:1428] T 68981913851243a7a1875fc99efb3c49 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.027s user 0.006s sys 0.000s
I20250114 02:54:35.458004 2406 raft_consensus.cc:2743] T a83bae91c29b463aa78412ce2e152ade P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:35.458789 1205 tablet_bootstrap.cc:492] T a2d68eccaaa64511a9ae36f2b07a7b59 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:35.459110 1206 raft_consensus.cc:357] T e06daadd346d45969ad5efcf03bc0870 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:35.459820 1206 raft_consensus.cc:383] T e06daadd346d45969ad5efcf03bc0870 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:35.461144 1206 raft_consensus.cc:738] T e06daadd346d45969ad5efcf03bc0870 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:35.461915 1206 consensus_queue.cc:260] T e06daadd346d45969ad5efcf03bc0870 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:35.471572 1207 tablet_bootstrap.cc:492] T 0b8361cd538046e0bcb4aac821748f67 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:35.472012 1207 ts_tablet_manager.cc:1397] T 0b8361cd538046e0bcb4aac821748f67 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.103s user 0.010s sys 0.000s
I20250114 02:54:35.476588 1207 raft_consensus.cc:357] T 0b8361cd538046e0bcb4aac821748f67 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:35.477260 1207 raft_consensus.cc:383] T 0b8361cd538046e0bcb4aac821748f67 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:35.477504 1207 raft_consensus.cc:738] T 0b8361cd538046e0bcb4aac821748f67 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:35.481684 1207 consensus_queue.cc:260] T 0b8361cd538046e0bcb4aac821748f67 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:35.485455 1207 ts_tablet_manager.cc:1428] T 0b8361cd538046e0bcb4aac821748f67 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.013s user 0.004s sys 0.000s
I20250114 02:54:35.500480 1206 ts_tablet_manager.cc:1428] T e06daadd346d45969ad5efcf03bc0870 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.043s user 0.004s sys 0.001s
I20250114 02:54:35.467857 1992 raft_consensus.cc:2238] T d05089eae2b74f9fb23a226d358a59da P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:35.501118 1992 raft_consensus.cc:2267] T d05089eae2b74f9fb23a226d358a59da P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:35.503124 1992 ts_tablet_manager.cc:1905] T d05089eae2b74f9fb23a226d358a59da P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:35.511281 2420 ts_tablet_manager.cc:1918] T 6e858a8449444ef999b9bbef612249d3 P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:35.511683 2420 log.cc:1198] T 6e858a8449444ef999b9bbef612249d3 P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/6e858a8449444ef999b9bbef612249d3
W20250114 02:54:35.514489 1040 consensus_peers.cc:487] T ed7f575653c848d7b076c17c38c4a244 P 90d6c2c58bb54e0eafb365625ac33871 -> Peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Couldn't send request to peer 525198593c8e47afaee54e74267b87e3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ed7f575653c848d7b076c17c38c4a244. This is attempt 1: this message will repeat every 5th retry.
I20250114 02:54:35.518416 2420 ts_tablet_manager.cc:1939] T 6e858a8449444ef999b9bbef612249d3 P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:35.521004 1207 tablet_bootstrap.cc:492] T 5d545ac27a7f49faace0655713405cea P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:35.521579 1205 tablet_bootstrap.cc:654] T a2d68eccaaa64511a9ae36f2b07a7b59 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:35.525488 2420 tablet_replica.cc:331] T d05089eae2b74f9fb23a226d358a59da P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:35.526449 1207 tablet_bootstrap.cc:654] T 5d545ac27a7f49faace0655713405cea P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:35.526403 2420 raft_consensus.cc:2238] T d05089eae2b74f9fb23a226d358a59da P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:35.526854 2420 raft_consensus.cc:2267] T d05089eae2b74f9fb23a226d358a59da P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:35.527817 1206 tablet_bootstrap.cc:492] T 797d6cdf36704c98aa106aac87a945ca P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:35.533303 2420 ts_tablet_manager.cc:1905] T d05089eae2b74f9fb23a226d358a59da P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:35.540268 1205 tablet_bootstrap.cc:492] T a2d68eccaaa64511a9ae36f2b07a7b59 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:35.540709 1205 ts_tablet_manager.cc:1397] T a2d68eccaaa64511a9ae36f2b07a7b59 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.082s user 0.008s sys 0.001s
I20250114 02:54:35.543434 1205 raft_consensus.cc:357] T a2d68eccaaa64511a9ae36f2b07a7b59 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:35.544047 1205 raft_consensus.cc:383] T a2d68eccaaa64511a9ae36f2b07a7b59 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:35.544291 1205 raft_consensus.cc:738] T a2d68eccaaa64511a9ae36f2b07a7b59 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:35.547425 1205 consensus_queue.cc:260] T a2d68eccaaa64511a9ae36f2b07a7b59 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:35.554229 1205 ts_tablet_manager.cc:1428] T a2d68eccaaa64511a9ae36f2b07a7b59 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.013s user 0.003s sys 0.002s
I20250114 02:54:35.555053 1205 tablet_bootstrap.cc:492] T d56255ca806d4e2cac09fbb5e1b1efbd P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:35.568477 2372 raft_consensus.cc:491] T ed109b49f20642f988161a3df7872419 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:35.568899 2372 raft_consensus.cc:513] T ed109b49f20642f988161a3df7872419 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:35.571019 2372 leader_election.cc:290] T ed109b49f20642f988161a3df7872419 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:35.572751 960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed109b49f20642f988161a3df7872419" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:35.573632 1207 tablet_bootstrap.cc:492] T 5d545ac27a7f49faace0655713405cea P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:35.573990 1207 ts_tablet_manager.cc:1397] T 5d545ac27a7f49faace0655713405cea P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.053s user 0.007s sys 0.000s
I20250114 02:54:35.573815 1100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed109b49f20642f988161a3df7872419" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
W20250114 02:54:35.575124 753 leader_election.cc:343] T ed109b49f20642f988161a3df7872419 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
W20250114 02:54:35.575829 751 leader_election.cc:343] T ed109b49f20642f988161a3df7872419 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:35.576150 751 leader_election.cc:304] T ed109b49f20642f988161a3df7872419 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:35.577096 2372 raft_consensus.cc:2743] T ed109b49f20642f988161a3df7872419 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:35.576190 1207 raft_consensus.cc:357] T 5d545ac27a7f49faace0655713405cea P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:35.579674 1207 raft_consensus.cc:383] T 5d545ac27a7f49faace0655713405cea P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:35.579913 1207 raft_consensus.cc:738] T 5d545ac27a7f49faace0655713405cea P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:35.584733 1206 tablet_bootstrap.cc:654] T 797d6cdf36704c98aa106aac87a945ca P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:35.592470 1207 consensus_queue.cc:260] T 5d545ac27a7f49faace0655713405cea P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:35.594944 1207 ts_tablet_manager.cc:1428] T 5d545ac27a7f49faace0655713405cea P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.021s user 0.006s sys 0.000s
I20250114 02:54:35.595979 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet 6e858a8449444ef999b9bbef612249d3 (table default.Table_5a9480c6474848d6b400e7dd274d15b0 [id=a17ad30ef32f4cbcb22b5a7d1c7096ff]) successfully deleted
W20250114 02:54:35.597440 456 catalog_manager.cc:4839] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet 3161f75bc4e84000af022faa9804c810 because the tablet was not found. No further retry: Not found: Tablet not found: 3161f75bc4e84000af022faa9804c810
W20250114 02:54:35.598404 456 catalog_manager.cc:4852] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet e285f86c29654b9697427516e99ca43a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e285f86c29654b9697427516e99ca43a already in progress: creating tablet
I20250114 02:54:35.600479 1207 tablet_bootstrap.cc:492] T a2d68eccaaa64511a9ae36f2b07a7b59 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:35.632051 1207 tablet_bootstrap.cc:654] T a2d68eccaaa64511a9ae36f2b07a7b59 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
W20250114 02:54:35.637140 1042 consensus_peers.cc:487] T ed7f575653c848d7b076c17c38c4a244 P 90d6c2c58bb54e0eafb365625ac33871 -> Peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Couldn't send request to peer 819f9e3412e54d44abb11d8b3ca877c2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ed7f575653c848d7b076c17c38c4a244. This is attempt 1: this message will repeat every 5th retry.
I20250114 02:54:35.648439 2372 raft_consensus.cc:491] T d56255ca806d4e2cac09fbb5e1b1efbd P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:35.648917 2372 raft_consensus.cc:513] T d56255ca806d4e2cac09fbb5e1b1efbd P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:35.650790 2372 leader_election.cc:290] T d56255ca806d4e2cac09fbb5e1b1efbd P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:35.652464 1103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d56255ca806d4e2cac09fbb5e1b1efbd" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
W20250114 02:54:35.653790 753 leader_election.cc:343] T d56255ca806d4e2cac09fbb5e1b1efbd P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:35.655612 960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d56255ca806d4e2cac09fbb5e1b1efbd" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
W20250114 02:54:35.658295 751 leader_election.cc:343] T d56255ca806d4e2cac09fbb5e1b1efbd P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:35.657361 468 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "default.Table_539d51840e36495ab6185647ab9044e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:54:35.658668 751 leader_election.cc:304] T d56255ca806d4e2cac09fbb5e1b1efbd P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:35.659668 2372 raft_consensus.cc:2743] T d56255ca806d4e2cac09fbb5e1b1efbd P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250114 02:54:35.659834 468 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_539d51840e36495ab6185647ab9044e3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:54:35.667575 1205 tablet_bootstrap.cc:654] T d56255ca806d4e2cac09fbb5e1b1efbd P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:35.674544 1207 tablet_bootstrap.cc:492] T a2d68eccaaa64511a9ae36f2b07a7b59 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:35.675004 1207 ts_tablet_manager.cc:1397] T a2d68eccaaa64511a9ae36f2b07a7b59 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.075s user 0.004s sys 0.004s
I20250114 02:54:35.676614 2420 ts_tablet_manager.cc:1918] T d05089eae2b74f9fb23a226d358a59da P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:35.677359 1207 raft_consensus.cc:357] T a2d68eccaaa64511a9ae36f2b07a7b59 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:35.678010 1207 raft_consensus.cc:383] T a2d68eccaaa64511a9ae36f2b07a7b59 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:35.678251 1207 raft_consensus.cc:738] T a2d68eccaaa64511a9ae36f2b07a7b59 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:35.678898 2420 log.cc:1198] T d05089eae2b74f9fb23a226d358a59da P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/d05089eae2b74f9fb23a226d358a59da
I20250114 02:54:35.678803 1207 consensus_queue.cc:260] T a2d68eccaaa64511a9ae36f2b07a7b59 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:35.681263 2420 ts_tablet_manager.cc:1939] T d05089eae2b74f9fb23a226d358a59da P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:35.685899 1206 tablet_bootstrap.cc:492] T 797d6cdf36704c98aa106aac87a945ca P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:35.686333 1206 ts_tablet_manager.cc:1397] T 797d6cdf36704c98aa106aac87a945ca P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.159s user 0.004s sys 0.005s
I20250114 02:54:35.689920 1207 ts_tablet_manager.cc:1428] T a2d68eccaaa64511a9ae36f2b07a7b59 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.015s user 0.006s sys 0.001s
I20250114 02:54:35.690738 1207 tablet_bootstrap.cc:492] T 670dc7ff15d644e18c296eeb2a6b99e1 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:35.692407 2406 raft_consensus.cc:491] T f40c66314342453b898da3ac0e7e86bf P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:35.692929 2406 raft_consensus.cc:513] T f40c66314342453b898da3ac0e7e86bf P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:35.695112 2406 leader_election.cc:290] T f40c66314342453b898da3ac0e7e86bf P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:35.697625 811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40c66314342453b898da3ac0e7e86bf" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:35.698508 811 raft_consensus.cc:2388] T f40c66314342453b898da3ac0e7e86bf P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 90d6c2c58bb54e0eafb365625ac33871 in current term 1: Already voted for candidate 525198593c8e47afaee54e74267b87e3 in this term.
I20250114 02:54:35.700042 1206 raft_consensus.cc:357] T 797d6cdf36704c98aa106aac87a945ca P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:35.700256 960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40c66314342453b898da3ac0e7e86bf" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:35.700722 1206 raft_consensus.cc:383] T 797d6cdf36704c98aa106aac87a945ca P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:35.700968 1206 raft_consensus.cc:738] T 797d6cdf36704c98aa106aac87a945ca P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:35.701683 1040 leader_election.cc:304] T f40c66314342453b898da3ac0e7e86bf P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90d6c2c58bb54e0eafb365625ac33871; no voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2
I20250114 02:54:35.702066 1206 consensus_queue.cc:260] T 797d6cdf36704c98aa106aac87a945ca P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:35.702759 2406 raft_consensus.cc:3054] T f40c66314342453b898da3ac0e7e86bf P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:35.712448 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet d05089eae2b74f9fb23a226d358a59da (table default.Table_5e1c3ceefda2466da137a659035f9ae4 [id=ee635d0ed76b46c9b2e139a75296017b]) successfully deleted
W20250114 02:54:35.714042 455 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 62af33f52a264fa8ba977b4a05e74d2b (table default.Table_20609e5296574025b1567d6463933b17 [id=b825f6b5a125445aa2feddab78584d73], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 62af33f52a264fa8ba977b4a05e74d2b consensus state has no leader
I20250114 02:54:35.680061 1992 ts_tablet_manager.cc:1918] T d05089eae2b74f9fb23a226d358a59da P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:35.716544 1992 log.cc:1198] T d05089eae2b74f9fb23a226d358a59da P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/d05089eae2b74f9fb23a226d358a59da
I20250114 02:54:35.717222 1992 ts_tablet_manager.cc:1939] T d05089eae2b74f9fb23a226d358a59da P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:35.719672 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet d05089eae2b74f9fb23a226d358a59da (table default.Table_5e1c3ceefda2466da137a659035f9ae4 [id=ee635d0ed76b46c9b2e139a75296017b]) successfully deleted
I20250114 02:54:35.728039 1207 tablet_bootstrap.cc:654] T 670dc7ff15d644e18c296eeb2a6b99e1 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:35.731334 1205 tablet_bootstrap.cc:492] T d56255ca806d4e2cac09fbb5e1b1efbd P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:35.736500 2406 raft_consensus.cc:2743] T f40c66314342453b898da3ac0e7e86bf P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:35.738775 1206 ts_tablet_manager.cc:1428] T 797d6cdf36704c98aa106aac87a945ca P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.052s user 0.007s sys 0.000s
I20250114 02:54:35.743770 1205 ts_tablet_manager.cc:1397] T d56255ca806d4e2cac09fbb5e1b1efbd P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.189s user 0.013s sys 0.007s
I20250114 02:54:35.746842 1205 raft_consensus.cc:357] T d56255ca806d4e2cac09fbb5e1b1efbd P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:35.755231 1206 tablet_bootstrap.cc:492] T 2e17f619ed2745078541cc1c924aff96 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:35.732935 2477 tablet.cc:1721] T 38fe8829d98547fb907ef78099df3d7f P 525198593c8e47afaee54e74267b87e3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250114 02:54:35.736732 2447 raft_consensus.cc:491] T 5d545ac27a7f49faace0655713405cea P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:35.766404 2447 raft_consensus.cc:513] T 5d545ac27a7f49faace0655713405cea P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:35.770346 2447 leader_election.cc:290] T 5d545ac27a7f49faace0655713405cea P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:35.764791 1205 raft_consensus.cc:383] T d56255ca806d4e2cac09fbb5e1b1efbd P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:35.773334 1205 raft_consensus.cc:738] T d56255ca806d4e2cac09fbb5e1b1efbd P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:35.757522 2449 consensus_queue.cc:1035] T f40c66314342453b898da3ac0e7e86bf P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:54:35.774580 1205 consensus_queue.cc:260] T d56255ca806d4e2cac09fbb5e1b1efbd P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:35.776876 1103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d545ac27a7f49faace0655713405cea" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:35.777429 1103 raft_consensus.cc:2463] T 5d545ac27a7f49faace0655713405cea P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:35.779093 813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d545ac27a7f49faace0655713405cea" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:35.779686 813 raft_consensus.cc:2463] T 5d545ac27a7f49faace0655713405cea P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:35.783615 429 master-stress-test.cc:579] Killed test thread 3/10
I20250114 02:54:35.783898 429 master-stress-test.cc:577] Killing test thread 4/10
I20250114 02:54:35.784178 429 master-stress-test.cc:579] Killed test thread 4/10
I20250114 02:54:35.784380 429 master-stress-test.cc:577] Killing test thread 5/10
I20250114 02:54:35.785501 1207 tablet_bootstrap.cc:492] T 670dc7ff15d644e18c296eeb2a6b99e1 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:35.785902 1207 ts_tablet_manager.cc:1397] T 670dc7ff15d644e18c296eeb2a6b99e1 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.095s user 0.007s sys 0.004s
I20250114 02:54:35.788051 1207 raft_consensus.cc:357] T 670dc7ff15d644e18c296eeb2a6b99e1 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:35.788846 1207 raft_consensus.cc:383] T 670dc7ff15d644e18c296eeb2a6b99e1 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:35.789106 1207 raft_consensus.cc:738] T 670dc7ff15d644e18c296eeb2a6b99e1 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:35.793210 1207 consensus_queue.cc:260] T 670dc7ff15d644e18c296eeb2a6b99e1 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:35.796306 1206 tablet_bootstrap.cc:654] T 2e17f619ed2745078541cc1c924aff96 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:35.800210 1207 ts_tablet_manager.cc:1428] T 670dc7ff15d644e18c296eeb2a6b99e1 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.014s user 0.006s sys 0.001s
I20250114 02:54:35.801793 898 leader_election.cc:304] T 5d545ac27a7f49faace0655713405cea P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:35.818877 2372 raft_consensus.cc:491] T e06daadd346d45969ad5efcf03bc0870 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:35.819478 2372 raft_consensus.cc:513] T e06daadd346d45969ad5efcf03bc0870 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:35.822551 2372 leader_election.cc:290] T e06daadd346d45969ad5efcf03bc0870 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:35.774430 470 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 141cede72b8741a8b0615d7c6bb823bb (table default.Table_4364d1df16d149d2ba10469ce7288d4e [id=13fc996d911b430290e37a94dd44b1ab]). Expected version 1 got 0
I20250114 02:54:35.803691 1205 ts_tablet_manager.cc:1428] T d56255ca806d4e2cac09fbb5e1b1efbd P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.059s user 0.003s sys 0.002s
I20250114 02:54:35.831600 1205 tablet_bootstrap.cc:492] T 0b8361cd538046e0bcb4aac821748f67 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:35.831421 797 tablet_service.cc:1467] Processing CreateTablet for tablet fd1aa3d4c6fb4bf2b1265c96b3177629 (DEFAULT_TABLE table=default.Table_dd50844b27ef42dda7cab6f276fac825 [id=af4dd8face234ca99bf7523004217964]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:35.832007 2449 raft_consensus.cc:2798] T 5d545ac27a7f49faace0655713405cea P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:35.832355 2449 raft_consensus.cc:491] T 5d545ac27a7f49faace0655713405cea P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:35.832576 797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd1aa3d4c6fb4bf2b1265c96b3177629. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:35.832664 2449 raft_consensus.cc:3054] T 5d545ac27a7f49faace0655713405cea P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:35.800012 2478 tablet.cc:1721] T 38fe8829d98547fb907ef78099df3d7f P 819f9e3412e54d44abb11d8b3ca877c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250114 02:54:35.841075 470 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 38fe8829d98547fb907ef78099df3d7f (table default.Table_60450873174a4c80aa7e2122343d82be [id=9299683349354f879847b7c9d93f6072]). Expected version 1 got 0
I20250114 02:54:35.842244 470 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet edc69b66bea54625a4b15867b75986b7 (table default.Table_0a0bda55a4a9476c98d961822b8a19bd [id=fb2be5c1ca9444a197f3315ea9c59b09]). Expected version 1 got 0
I20250114 02:54:35.785413 468 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "default.Table_7416825652064407ab1bc5469bc21167" } new_table_name: "default.Table_6d9d09d06ff34d9197882e2f88179301" modify_external_catalogs: true
I20250114 02:54:35.852258 1207 tablet_bootstrap.cc:492] T 68981913851243a7a1875fc99efb3c49 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:35.854252 960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06daadd346d45969ad5efcf03bc0870" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:35.857827 934 tablet_service.cc:1467] Processing CreateTablet for tablet fd1aa3d4c6fb4bf2b1265c96b3177629 (DEFAULT_TABLE table=default.Table_dd50844b27ef42dda7cab6f276fac825 [id=af4dd8face234ca99bf7523004217964]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:35.859241 934 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd1aa3d4c6fb4bf2b1265c96b3177629. 1 dirs total, 0 dirs full, 0 dirs failed
W20250114 02:54:35.860612 751 leader_election.cc:343] T e06daadd346d45969ad5efcf03bc0870 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:35.860642 1075 tablet_service.cc:1467] Processing CreateTablet for tablet fd1aa3d4c6fb4bf2b1265c96b3177629 (DEFAULT_TABLE table=default.Table_dd50844b27ef42dda7cab6f276fac825 [id=af4dd8face234ca99bf7523004217964]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:35.881981 1205 tablet_bootstrap.cc:654] T 0b8361cd538046e0bcb4aac821748f67 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:35.882722 1206 tablet_bootstrap.cc:492] T 2e17f619ed2745078541cc1c924aff96 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:35.883101 1206 ts_tablet_manager.cc:1397] T 2e17f619ed2745078541cc1c924aff96 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.128s user 0.005s sys 0.003s
I20250114 02:54:35.893839 1206 raft_consensus.cc:357] T 2e17f619ed2745078541cc1c924aff96 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:35.894558 1206 raft_consensus.cc:383] T 2e17f619ed2745078541cc1c924aff96 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:35.894873 1206 raft_consensus.cc:738] T 2e17f619ed2745078541cc1c924aff96 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:35.895628 1206 consensus_queue.cc:260] T 2e17f619ed2745078541cc1c924aff96 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:35.913818 1206 ts_tablet_manager.cc:1428] T 2e17f619ed2745078541cc1c924aff96 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.030s user 0.007s sys 0.001s
I20250114 02:54:35.914768 1206 tablet_bootstrap.cc:492] T 97de018391f84b0ebd3130861681b7b5 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:35.920626 1075 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd1aa3d4c6fb4bf2b1265c96b3177629. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:35.921921 939 tablet_service.cc:1467] Processing CreateTablet for tablet 811205d3eab043e2862d886c11a7201a (DEFAULT_TABLE table=default.Table_4150850ec6774c4a82d174648a35d1b4 [id=180820378e5f4a0899e6e24aa949b8fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:35.922935 939 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 811205d3eab043e2862d886c11a7201a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:35.924273 1103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06daadd346d45969ad5efcf03bc0870" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:35.932442 1207 tablet_bootstrap.cc:654] T 68981913851243a7a1875fc99efb3c49 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:35.935664 1084 tablet_service.cc:1467] Processing CreateTablet for tablet 811205d3eab043e2862d886c11a7201a (DEFAULT_TABLE table=default.Table_4150850ec6774c4a82d174648a35d1b4 [id=180820378e5f4a0899e6e24aa949b8fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:35.936712 1084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 811205d3eab043e2862d886c11a7201a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:35.940989 932 tablet_service.cc:1467] Processing CreateTablet for tablet 4f565a17152243e5a122a032879e0604 (DEFAULT_TABLE table=default.Table_4126dce6d1614249887aee151c0b1085 [id=8d94afa1935a428ba36c8171805cfa9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:35.942114 932 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f565a17152243e5a122a032879e0604. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:35.944442 1085 tablet_service.cc:1467] Processing CreateTablet for tablet 4f565a17152243e5a122a032879e0604 (DEFAULT_TABLE table=default.Table_4126dce6d1614249887aee151c0b1085 [id=8d94afa1935a428ba36c8171805cfa9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:35.945461 1085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f565a17152243e5a122a032879e0604. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:35.947335 797 tablet_service.cc:1467] Processing CreateTablet for tablet 811205d3eab043e2862d886c11a7201a (DEFAULT_TABLE table=default.Table_4150850ec6774c4a82d174648a35d1b4 [id=180820378e5f4a0899e6e24aa949b8fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:35.948308 797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 811205d3eab043e2862d886c11a7201a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:35.951457 785 tablet_service.cc:1467] Processing CreateTablet for tablet 4f565a17152243e5a122a032879e0604 (DEFAULT_TABLE table=default.Table_4126dce6d1614249887aee151c0b1085 [id=8d94afa1935a428ba36c8171805cfa9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:35.952503 785 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f565a17152243e5a122a032879e0604. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:35.976892 1207 tablet_bootstrap.cc:492] T 68981913851243a7a1875fc99efb3c49 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:35.977430 1207 ts_tablet_manager.cc:1397] T 68981913851243a7a1875fc99efb3c49 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.177s user 0.008s sys 0.003s
I20250114 02:54:35.843591 470 catalog_manager.cc:5526] T 967f4d5b944543d49e87645d069b5b55 P 525198593c8e47afaee54e74267b87e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 525198593c8e47afaee54e74267b87e3 (127.0.107.66). New cstate: current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:35.980108 1207 raft_consensus.cc:357] T 68981913851243a7a1875fc99efb3c49 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:35.980861 1207 raft_consensus.cc:383] T 68981913851243a7a1875fc99efb3c49 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:35.981163 1207 raft_consensus.cc:738] T 68981913851243a7a1875fc99efb3c49 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:35.981891 1207 consensus_queue.cc:260] T 68981913851243a7a1875fc99efb3c49 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
W20250114 02:54:35.985846 898 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.67:45749, user_credentials={real_user=slave}} blocked reactor thread for 92426.3us
W20250114 02:54:35.994343 753 leader_election.cc:343] T e06daadd346d45969ad5efcf03bc0870 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:35.995049 753 leader_election.cc:304] T e06daadd346d45969ad5efcf03bc0870 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:35.996573 2372 raft_consensus.cc:2743] T e06daadd346d45969ad5efcf03bc0870 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:36.005584 1206 tablet_bootstrap.cc:654] T 97de018391f84b0ebd3130861681b7b5 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:36.014721 2447 consensus_queue.cc:1035] T f40c66314342453b898da3ac0e7e86bf P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:36.015483 2389 consensus_queue.cc:1035] T d2a7b2bee5c1440fb705b172c18a9247 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:36.025600 1207 ts_tablet_manager.cc:1428] T 68981913851243a7a1875fc99efb3c49 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.048s user 0.005s sys 0.000s
I20250114 02:54:36.026615 1207 tablet_bootstrap.cc:492] T be7aad41ccdd4ef9bcfe8a985155708f P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:36.042940 2491 tablet.cc:1721] T 38fe8829d98547fb907ef78099df3d7f P 90d6c2c58bb54e0eafb365625ac33871: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250114 02:54:35.981549 470 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet a52d6749ff834b62a8e20a2f45935240 (table default.Table_1c9b7896941e4289b1c12e20c210c2c1 [id=b3ee15b9f10f4d489edbde3af9034d25]). Expected version 1 got 0
I20250114 02:54:36.057518 1205 tablet_bootstrap.cc:492] T 0b8361cd538046e0bcb4aac821748f67 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:36.058045 1205 ts_tablet_manager.cc:1397] T 0b8361cd538046e0bcb4aac821748f67 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.227s user 0.012s sys 0.008s
I20250114 02:54:36.060730 1205 raft_consensus.cc:357] T 0b8361cd538046e0bcb4aac821748f67 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:36.069461 470 catalog_manager.cc:5526] T 719188f639724851a5fc2b37cb60fdbb P 525198593c8e47afaee54e74267b87e3 reported cstate change: term changed from 0 to 2, leader changed from <none> to 525198593c8e47afaee54e74267b87e3 (127.0.107.66). New cstate: current_term: 2 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:36.070704 470 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 719188f639724851a5fc2b37cb60fdbb (table default.Table_8f8a18d7868c44bbb4d66a0692b08186 [id=e6292728826f437cb75c524a8aef2e87]). Expected version 1 got 0
I20250114 02:54:36.074646 1207 tablet_bootstrap.cc:654] T be7aad41ccdd4ef9bcfe8a985155708f P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:36.075692 1205 raft_consensus.cc:383] T 0b8361cd538046e0bcb4aac821748f67 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:36.077656 1205 raft_consensus.cc:738] T 0b8361cd538046e0bcb4aac821748f67 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:36.082697 1205 consensus_queue.cc:260] T 0b8361cd538046e0bcb4aac821748f67 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:36.003325 2449 raft_consensus.cc:513] T 5d545ac27a7f49faace0655713405cea P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:36.094525 1099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d545ac27a7f49faace0655713405cea" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:36.095079 1099 raft_consensus.cc:3054] T 5d545ac27a7f49faace0655713405cea P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:36.098133 2389 consensus_queue.cc:1035] T d2a7b2bee5c1440fb705b172c18a9247 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:36.101560 1099 raft_consensus.cc:2463] T 5d545ac27a7f49faace0655713405cea P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:36.101862 2484 raft_consensus.cc:491] T 797d6cdf36704c98aa106aac87a945ca P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:36.102296 2484 raft_consensus.cc:513] T 797d6cdf36704c98aa106aac87a945ca P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:36.102571 898 leader_election.cc:304] T 5d545ac27a7f49faace0655713405cea P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:36.103363 2447 raft_consensus.cc:2798] T 5d545ac27a7f49faace0655713405cea P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:36.103793 2447 raft_consensus.cc:695] T 5d545ac27a7f49faace0655713405cea P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Becoming Leader. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Running, Role: LEADER
I20250114 02:54:36.104528 2486 raft_consensus.cc:491] T d56255ca806d4e2cac09fbb5e1b1efbd P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:36.105003 2484 leader_election.cc:290] T 797d6cdf36704c98aa106aac87a945ca P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:36.104920 2486 raft_consensus.cc:513] T d56255ca806d4e2cac09fbb5e1b1efbd P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:36.105085 2447 consensus_queue.cc:237] T 5d545ac27a7f49faace0655713405cea P 525198593c8e47afaee54e74267b87e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:36.106354 1206 tablet_bootstrap.cc:492] T 97de018391f84b0ebd3130861681b7b5 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:36.106822 1206 ts_tablet_manager.cc:1397] T 97de018391f84b0ebd3130861681b7b5 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.192s user 0.019s sys 0.003s
I20250114 02:54:36.106786 2486 leader_election.cc:290] T d56255ca806d4e2cac09fbb5e1b1efbd P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:36.109488 1099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d56255ca806d4e2cac09fbb5e1b1efbd" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:36.111685 959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "797d6cdf36704c98aa106aac87a945ca" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
W20250114 02:54:36.112843 898 leader_election.cc:343] T d56255ca806d4e2cac09fbb5e1b1efbd P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
W20250114 02:54:36.112824 751 leader_election.cc:343] T 797d6cdf36704c98aa106aac87a945ca P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:36.115025 1099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "797d6cdf36704c98aa106aac87a945ca" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:36.115180 811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d56255ca806d4e2cac09fbb5e1b1efbd" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:36.115697 811 raft_consensus.cc:2463] T d56255ca806d4e2cac09fbb5e1b1efbd P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
W20250114 02:54:36.120854 458 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.0.107.67:45749, user_credentials={real_user=slave}} blocked reactor thread for 144710us
I20250114 02:54:36.130510 2449 leader_election.cc:290] T 5d545ac27a7f49faace0655713405cea P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Requested vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
W20250114 02:54:36.133405 753 leader_election.cc:343] T 797d6cdf36704c98aa106aac87a945ca P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:36.137579 1206 raft_consensus.cc:357] T 97de018391f84b0ebd3130861681b7b5 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:36.139338 1206 raft_consensus.cc:383] T 97de018391f84b0ebd3130861681b7b5 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:36.140517 1206 raft_consensus.cc:738] T 97de018391f84b0ebd3130861681b7b5 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:36.141099 1206 consensus_queue.cc:260] T 97de018391f84b0ebd3130861681b7b5 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:36.169543 2351 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b303b7df086340a0b5ad6373940ae750. Latest consensus state: current_term: 1 leader_uuid: "b303b7df086340a0b5ad6373940ae750" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b303b7df086340a0b5ad6373940ae750" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 45367 } } peers { permanent_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } } peers { permanent_uuid: "87b3db09ac87436e9a60d6de5fedaff5" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } } }
I20250114 02:54:36.170329 2351 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [sys.catalog]: This master's current role is: FOLLOWER
I20250114 02:54:36.172408 1206 ts_tablet_manager.cc:1428] T 97de018391f84b0ebd3130861681b7b5 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.065s user 0.004s sys 0.003s
I20250114 02:54:36.173207 753 leader_election.cc:304] T 797d6cdf36704c98aa106aac87a945ca P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:36.174667 2484 raft_consensus.cc:2743] T 797d6cdf36704c98aa106aac87a945ca P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:36.176072 1207 tablet_bootstrap.cc:492] T be7aad41ccdd4ef9bcfe8a985155708f P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:36.177906 1207 ts_tablet_manager.cc:1397] T be7aad41ccdd4ef9bcfe8a985155708f P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.151s user 0.009s sys 0.001s
I20250114 02:54:36.180176 1207 raft_consensus.cc:357] T be7aad41ccdd4ef9bcfe8a985155708f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:36.180841 1207 raft_consensus.cc:383] T be7aad41ccdd4ef9bcfe8a985155708f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:36.181075 1207 raft_consensus.cc:738] T be7aad41ccdd4ef9bcfe8a985155708f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:36.183458 2406 raft_consensus.cc:491] T 670dc7ff15d644e18c296eeb2a6b99e1 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:36.183878 2406 raft_consensus.cc:513] T 670dc7ff15d644e18c296eeb2a6b99e1 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:36.186097 2422 raft_consensus.cc:491] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b303b7df086340a0b5ad6373940ae750)
I20250114 02:54:36.186709 2422 raft_consensus.cc:513] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b303b7df086340a0b5ad6373940ae750" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 45367 } } peers { permanent_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } } peers { permanent_uuid: "87b3db09ac87436e9a60d6de5fedaff5" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } }
I20250114 02:54:36.188796 2406 leader_election.cc:290] T 670dc7ff15d644e18c296eeb2a6b99e1 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:36.131103 1205 ts_tablet_manager.cc:1428] T 0b8361cd538046e0bcb4aac821748f67 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.073s user 0.002s sys 0.003s
I20250114 02:54:36.190217 1205 tablet_bootstrap.cc:492] T 612987f3e4ea4a02932ceefa0d740425 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:36.193933 813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "670dc7ff15d644e18c296eeb2a6b99e1" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:36.194522 813 raft_consensus.cc:2463] T 670dc7ff15d644e18c296eeb2a6b99e1 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 0.
I20250114 02:54:36.195504 1042 leader_election.cc:304] T 670dc7ff15d644e18c296eeb2a6b99e1 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:36.196208 2406 raft_consensus.cc:2798] T 670dc7ff15d644e18c296eeb2a6b99e1 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:36.196574 2406 raft_consensus.cc:491] T 670dc7ff15d644e18c296eeb2a6b99e1 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:36.196854 2406 raft_consensus.cc:3054] T 670dc7ff15d644e18c296eeb2a6b99e1 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:36.196843 960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "670dc7ff15d644e18c296eeb2a6b99e1" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
W20250114 02:54:36.198271 1040 leader_election.cc:343] T 670dc7ff15d644e18c296eeb2a6b99e1 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:36.200851 1206 tablet_bootstrap.cc:492] T 45301f8823ed4ec494e792d41f14f71f P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
W20250114 02:54:36.179975 2276 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.67:41121 (request call id 10) took 2379 ms (client timeout 1000 ms). Trace:
I20250114 02:54:36.210363 1207 consensus_queue.cc:260] T be7aad41ccdd4ef9bcfe8a985155708f P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:36.221190 813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d545ac27a7f49faace0655713405cea" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:36.221777 813 raft_consensus.cc:3054] T 5d545ac27a7f49faace0655713405cea P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
W20250114 02:54:36.203022 2276 rpcz_store.cc:269] 0114 02:54:33.800891 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:33.801157 (+ 266us) service_pool.cc:225] Handling call
0114 02:54:36.176839 (+2375682us) spinlock_profiling.cc:243] Waited 2.37 s on lock 0x7b54000104d4. stack: 00007f2fec450f63 00007f2fec450e60 00007f2febfc7584 00000000004fcd24 00000000004fccb9 00007f2ff3351d50 00007f2ff33d4f2a 00007f2ff5079dec 00007f2ff5171dee 00007f2ff2ecffc4 00007f2ff2ecff52 00007f2ff2ecfe81 00007f2ff2ecfdfc 00007f2ff2ecf0b2 00007f2ff1199dfc 00007f2ff1199236
0114 02:54:36.179756 (+ 2917us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5457966848}
I20250114 02:54:36.225111 2259 rpcz_store.cc:275] Call kudu.master.MasterService.GetMasterRegistration from 127.0.0.1:60438 (request call id 0) took 1042 ms. Trace:
I20250114 02:54:36.226644 1207 ts_tablet_manager.cc:1428] T be7aad41ccdd4ef9bcfe8a985155708f P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.048s user 0.005s sys 0.000s
I20250114 02:54:36.225494 2259 rpcz_store.cc:276] 0114 02:54:35.170464 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:35.170644 (+ 180us) service_pool.cc:225] Handling call
0114 02:54:36.209001 (+1038357us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b54000104d4. stack: 00007f2fec450f63 00007f2fec450e60 00007f2febfc7584 00000000004fcd24 00000000004fccb9 00007f2ff3351d50 00007f2ff33d4f2a 00007f2ff5079dec 00007f2ff5174dea 00007f2ff2f05e47 00007f2ff2f05dd2 00007f2ff2f05d01 00007f2ff2f05c7c 00007f2ff2f04f32 00007f2ff1199dfc 00007f2ff1199236
0114 02:54:36.212352 (+ 3351us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2341071616}
I20250114 02:54:36.227492 1207 tablet_bootstrap.cc:492] T 612987f3e4ea4a02932ceefa0d740425 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:36.227488 555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "87b3db09ac87436e9a60d6de5fedaff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 406 } } ignore_live_leader: false dest_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" is_pre_election: true
I20250114 02:54:36.233500 1206 tablet_bootstrap.cc:654] T 45301f8823ed4ec494e792d41f14f71f P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
W20250114 02:54:36.236671 2271 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.67:41121 (request call id 11) took 1382 ms (client timeout 1000 ms). Trace:
W20250114 02:54:36.237195 2271 rpcz_store.cc:269] 0114 02:54:34.835818 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:34.836072 (+ 254us) service_pool.cc:225] Handling call
0114 02:54:36.213174 (+1377102us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b54000104d4. stack: 00007f2fec450f63 00007f2fec450e60 00007f2febfc7584 00000000004fcd24 00000000004fccb9 00007f2ff3351d50 00007f2ff33d4f2a 00007f2ff5079dec 00007f2ff5171dee 00007f2ff2ecffc4 00007f2ff2ecff52 00007f2ff2ecfe81 00007f2ff2ecfdfc 00007f2ff2ecf0b2 00007f2ff1199dfc 00007f2ff1199236
0114 02:54:36.217472 (+ 4298us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3101108096}
W20250114 02:54:36.239092 2272 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.66:60145 (request call id 8) took 2393 ms (client timeout 1000 ms). Trace:
I20250114 02:54:36.209551 2406 raft_consensus.cc:513] T 670dc7ff15d644e18c296eeb2a6b99e1 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:36.244056 811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "670dc7ff15d644e18c296eeb2a6b99e1" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:36.243527 470 catalog_manager.cc:5526] T 30a56fb2d941470590e17f37b90aef26 P 525198593c8e47afaee54e74267b87e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 525198593c8e47afaee54e74267b87e3 (127.0.107.66). New cstate: current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:36.244614 811 raft_consensus.cc:3054] T 670dc7ff15d644e18c296eeb2a6b99e1 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:36.244810 470 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 30a56fb2d941470590e17f37b90aef26 (table default.Table_0e0710e3f43e4e859ad32662704ee41c [id=66082998119e4c43a666fb43087fc622]). Expected version 1 got 0
I20250114 02:54:36.245448 470 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet df045d80d8ce4ff194a6d2039a44bce6 (table default.Table_2befe573a994473abe0556ba7e713686 [id=da0258578bb34ae29551fc593bf7d0b5]). Expected version 1 got 0
I20250114 02:54:36.245644 1205 tablet_bootstrap.cc:654] T 612987f3e4ea4a02932ceefa0d740425 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:36.170023 2447 raft_consensus.cc:491] T 68981913851243a7a1875fc99efb3c49 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:36.248330 2447 raft_consensus.cc:513] T 68981913851243a7a1875fc99efb3c49 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:36.250546 2447 leader_election.cc:290] T 68981913851243a7a1875fc99efb3c49 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:36.175689 898 leader_election.cc:304] T d56255ca806d4e2cac09fbb5e1b1efbd P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters: 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:36.252524 960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "670dc7ff15d644e18c296eeb2a6b99e1" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
W20250114 02:54:36.252681 898 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.65:39397, user_credentials={real_user=slave}} blocked reactor thread for 77084.8us
W20250114 02:54:36.253861 1040 leader_election.cc:343] T 670dc7ff15d644e18c296eeb2a6b99e1 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:36.253924 2143 raft_consensus.cc:2798] T d56255ca806d4e2cac09fbb5e1b1efbd P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:36.254230 2143 raft_consensus.cc:491] T d56255ca806d4e2cac09fbb5e1b1efbd P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:36.254400 2406 leader_election.cc:290] T 670dc7ff15d644e18c296eeb2a6b99e1 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Requested vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:36.254498 2143 raft_consensus.cc:3054] T d56255ca806d4e2cac09fbb5e1b1efbd P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:36.255250 1103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68981913851243a7a1875fc99efb3c49" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:36.255805 1103 raft_consensus.cc:2463] T 68981913851243a7a1875fc99efb3c49 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:36.259315 1207 tablet_bootstrap.cc:654] T 612987f3e4ea4a02932ceefa0d740425 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:36.263067 2143 raft_consensus.cc:513] T d56255ca806d4e2cac09fbb5e1b1efbd P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
W20250114 02:54:36.264318 2260 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.66:60145 (request call id 9) took 1255 ms (client timeout 1000 ms). Trace:
W20250114 02:54:36.265605 2260 rpcz_store.cc:269] 0114 02:54:34.966889 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:34.972342 (+ 5453us) service_pool.cc:225] Handling call
0114 02:54:36.192748 (+1220406us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b54000104d4. stack: 00007f2fec450f63 00007f2fec450e60 00007f2febfc7584 00000000004fcd24 00000000004fccb9 00007f2ff3351d50 00007f2ff33d4f2a 00007f2ff5079dec 00007f2ff5171dee 00007f2ff2ecffc4 00007f2ff2ecff52 00007f2ff2ecfe81 00007f2ff2ecfdfc 00007f2ff2ecf0b2 00007f2ff1199dfc 00007f2ff1199236
0114 02:54:36.222616 (+ 29868us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2788982016}
I20250114 02:54:36.269286 898 leader_election.cc:304] T 68981913851243a7a1875fc99efb3c49 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:36.270023 811 raft_consensus.cc:2463] T 670dc7ff15d644e18c296eeb2a6b99e1 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1.
I20250114 02:54:36.270927 1103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d56255ca806d4e2cac09fbb5e1b1efbd" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:36.271235 1042 leader_election.cc:304] T 670dc7ff15d644e18c296eeb2a6b99e1 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters: 525198593c8e47afaee54e74267b87e3
I20250114 02:54:36.272473 2449 raft_consensus.cc:2798] T 68981913851243a7a1875fc99efb3c49 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:36.272818 2449 raft_consensus.cc:491] T 68981913851243a7a1875fc99efb3c49 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:36.273139 2449 raft_consensus.cc:3054] T 68981913851243a7a1875fc99efb3c49 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
W20250114 02:54:36.274345 2281 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.65:44399 (request call id 8) took 2473 ms (client timeout 1000 ms). Trace:
W20250114 02:54:36.274698 2281 rpcz_store.cc:269] 0114 02:54:33.801330 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:33.801548 (+ 218us) service_pool.cc:225] Handling call
0114 02:54:36.181158 (+2379610us) spinlock_profiling.cc:243] Waited 2.36 s on lock 0x7b54000104d4. stack: 00007f2fec450f63 00007f2fec450e60 00007f2febfc7584 00000000004fcd24 00000000004fccb9 00007f2ff3351d50 00007f2ff33d4f2a 00007f2ff5079dec 00007f2ff5171dee 00007f2ff2ecffc4 00007f2ff2ecff52 00007f2ff2ecfe81 00007f2ff2ecfdfc 00007f2ff2ecf0b2 00007f2ff1199dfc 00007f2ff1199236
0114 02:54:36.274174 (+ 93016us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5438034688}
I20250114 02:54:36.275959 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68981913851243a7a1875fc99efb3c49" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:36.276554 818 raft_consensus.cc:2463] T 68981913851243a7a1875fc99efb3c49 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:36.226200 2422 leader_election.cc:290] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b303b7df086340a0b5ad6373940ae750 (127.0.107.126:45367), 687d4ff6d3cb42cf90a965070b0d8a70 (127.0.107.125:46837)
W20250114 02:54:36.276607 898 leader_election.cc:343] T d56255ca806d4e2cac09fbb5e1b1efbd P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
W20250114 02:54:36.277175 2270 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.65:44399 (request call id 9) took 1296 ms (client timeout 1000 ms). Trace:
W20250114 02:54:36.277936 2270 rpcz_store.cc:269] 0114 02:54:34.911667 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:34.911892 (+ 225us) service_pool.cc:225] Handling call
0114 02:54:36.195851 (+1283959us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b54000104d4. stack: 00007f2fec450f63 00007f2fec450e60 00007f2febfc7584 00000000004fcd24 00000000004fccb9 00007f2ff3351d50 00007f2ff33d4f2a 00007f2ff5079dec 00007f2ff5171dee 00007f2ff2ecffc4 00007f2ff2ecff52 00007f2ff2ecfe81 00007f2ff2ecfdfc 00007f2ff2ecf0b2 00007f2ff1199dfc 00007f2ff1199236
0114 02:54:36.207513 (+ 11662us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2927247232}
W20250114 02:54:36.239493 2272 rpcz_store.cc:269] 0114 02:54:33.828034 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:33.828214 (+ 180us) service_pool.cc:225] Handling call
0114 02:54:36.212535 (+2384321us) spinlock_profiling.cc:243] Waited 2.36 s on lock 0x7b54000104d4. stack: 00007f2fec450f63 00007f2fec450e60 00007f2febfc7584 00000000004fcd24 00000000004fccb9 00007f2ff3351d50 00007f2ff33d4f2a 00007f2ff5079dec 00007f2ff5171dee 00007f2ff2ecffc4 00007f2ff2ecff52 00007f2ff2ecfe81 00007f2ff2ecfdfc 00007f2ff2ecf0b2 00007f2ff1199dfc 00007f2ff1199236
0114 02:54:36.221447 (+ 8912us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5423348480}
I20250114 02:54:36.280441 2406 raft_consensus.cc:2798] T 670dc7ff15d644e18c296eeb2a6b99e1 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:36.280884 2406 raft_consensus.cc:695] T 670dc7ff15d644e18c296eeb2a6b99e1 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Becoming Leader. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Running, Role: LEADER
I20250114 02:54:36.287554 2486 consensus_queue.cc:1035] T a83bae91c29b463aa78412ce2e152ade P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:36.289150 2406 consensus_queue.cc:237] T 670dc7ff15d644e18c296eeb2a6b99e1 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:36.290129 813 raft_consensus.cc:2463] T 5d545ac27a7f49faace0655713405cea P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:36.296175 1206 tablet_bootstrap.cc:492] T 45301f8823ed4ec494e792d41f14f71f P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:36.296713 1206 ts_tablet_manager.cc:1397] T 45301f8823ed4ec494e792d41f14f71f P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.096s user 0.008s sys 0.001s
I20250114 02:54:36.296664 2496 consensus_queue.cc:1035] T 0b8989d340fa4ae992ef97e226d41110 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:36.297250 2143 leader_election.cc:290] T d56255ca806d4e2cac09fbb5e1b1efbd P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Requested vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:36.307024 1205 tablet_bootstrap.cc:492] T 612987f3e4ea4a02932ceefa0d740425 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:36.307699 1205 ts_tablet_manager.cc:1397] T 612987f3e4ea4a02932ceefa0d740425 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.118s user 0.010s sys 0.001s
I20250114 02:54:36.312148 1205 raft_consensus.cc:357] T 612987f3e4ea4a02932ceefa0d740425 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:36.314085 1206 raft_consensus.cc:357] T 45301f8823ed4ec494e792d41f14f71f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:36.314741 1206 raft_consensus.cc:383] T 45301f8823ed4ec494e792d41f14f71f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:36.315021 1206 raft_consensus.cc:738] T 45301f8823ed4ec494e792d41f14f71f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:36.315675 1206 consensus_queue.cc:260] T 45301f8823ed4ec494e792d41f14f71f P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:36.321736 2484 raft_consensus.cc:491] T 45301f8823ed4ec494e792d41f14f71f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:36.322118 2484 raft_consensus.cc:513] T 45301f8823ed4ec494e792d41f14f71f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:36.324148 2484 leader_election.cc:290] T 45301f8823ed4ec494e792d41f14f71f P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:36.326921 1207 tablet_bootstrap.cc:492] T 612987f3e4ea4a02932ceefa0d740425 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:36.328192 1207 ts_tablet_manager.cc:1397] T 612987f3e4ea4a02932ceefa0d740425 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.101s user 0.009s sys 0.000s
I20250114 02:54:36.334499 1206 ts_tablet_manager.cc:1428] T 45301f8823ed4ec494e792d41f14f71f P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.037s user 0.003s sys 0.002s
I20250114 02:54:36.335474 1206 tablet_bootstrap.cc:492] T e7ca0bd1dbd542de9ee11e6c72522ae0 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:36.340270 1207 raft_consensus.cc:357] T 612987f3e4ea4a02932ceefa0d740425 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:36.340909 1207 raft_consensus.cc:383] T 612987f3e4ea4a02932ceefa0d740425 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:36.341154 1207 raft_consensus.cc:738] T 612987f3e4ea4a02932ceefa0d740425 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:36.341696 1207 consensus_queue.cc:260] T 612987f3e4ea4a02932ceefa0d740425 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
W20250114 02:54:36.354422 468 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5c26a33248d34bd2bfa0455249fa8156 consensus state has no leader
I20250114 02:54:36.354986 468 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:33540 (request call id 333) took 1020 ms. Trace:
I20250114 02:54:36.355479 1206 tablet_bootstrap.cc:654] T e7ca0bd1dbd542de9ee11e6c72522ae0 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:36.355295 468 rpcz_store.cc:276] 0114 02:54:35.334238 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:35.784992 (+450754us) service_pool.cc:225] Handling call
0114 02:54:35.849016 (+ 64024us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0114 02:54:35.849824 (+ 808us) catalog_manager.cc:3627] Apply alter schema
0114 02:54:35.850442 (+ 618us) catalog_manager.cc:3659] Acquired catalog manager lock
0114 02:54:35.850752 (+ 310us) catalog_manager.cc:3817] Updating metadata on disk
0114 02:54:35.976733 (+125981us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:35.976780 (+ 47us) write_op.cc:454] Released schema lock
0114 02:54:35.977130 (+ 350us) catalog_manager.cc:3848] Committing alterations to in-memory state
0114 02:54:36.354903 (+377773us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:35.944539 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:35.944918 (+ 379us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:35.944942 (+ 24us) write_op.cc:435] Acquired schema lock
0114 02:54:35.944953 (+ 11us) tablet.cc:659] Decoding operations
0114 02:54:35.945216 (+ 263us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:35.945256 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:35.945276 (+ 20us) tablet.cc:682] Acquiring locks for 1 operations
0114 02:54:35.945421 (+ 145us) tablet.cc:698] Row locks acquired
0114 02:54:35.945435 (+ 14us) write_op.cc:260] PREPARE: finished
0114 02:54:35.945546 (+ 111us) write_op.cc:270] Start()
0114 02:54:35.945638 (+ 92us) write_op.cc:276] Timestamp: P: 1736823275945415 usec, L: 0
0114 02:54:35.945653 (+ 15us) op_driver.cc:348] REPLICATION: starting
0114 02:54:35.946119 (+ 466us) log.cc:844] Serialized 405 byte log entry
0114 02:54:35.962094 (+ 15975us) op_driver.cc:464] REPLICATION: finished
0114 02:54:35.962416 (+ 322us) write_op.cc:301] APPLY: starting
0114 02:54:35.962486 (+ 70us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:35.962570 (+ 84us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:35.962698 (+ 128us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:35.962878 (+ 180us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:35.963015 (+ 137us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:35.963053 (+ 38us) write_op.cc:312] APPLY: finished
0114 02:54:35.963223 (+ 170us) log.cc:844] Serialized 23 byte log entry
0114 02:54:35.963353 (+ 130us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:35.963415 (+ 62us) write_op.cc:454] Released schema lock
0114 02:54:35.963443 (+ 28us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":187,"apply.run_cpu_time_us":1191,"apply.run_wall_time_us":1190,"num_ops":1,"prepare.queue_time_us":92502,"prepare.run_cpu_time_us":1886,"prepare.run_wall_time_us":1886,"raft.queue_time_us":1166,"raft.run_cpu_time_us":598,"raft.run_wall_time_us":599,"replication_time_us":16350}]]}
I20250114 02:54:36.365064 2406 raft_consensus.cc:491] T 0b8361cd538046e0bcb4aac821748f67 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:36.365532 2406 raft_consensus.cc:513] T 0b8361cd538046e0bcb4aac821748f67 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:36.366506 1103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45301f8823ed4ec494e792d41f14f71f" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:36.344086 2295 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60424 (request call id 0) took 2688 ms. Trace:
I20250114 02:54:36.367290 2406 leader_election.cc:290] T 0b8361cd538046e0bcb4aac821748f67 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:36.367844 813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d56255ca806d4e2cac09fbb5e1b1efbd" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:36.369935 813 raft_consensus.cc:3054] T d56255ca806d4e2cac09fbb5e1b1efbd P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:36.370895 1207 ts_tablet_manager.cc:1428] T 612987f3e4ea4a02932ceefa0d740425 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.042s user 0.004s sys 0.000s
I20250114 02:54:36.317014 2449 raft_consensus.cc:513] T 68981913851243a7a1875fc99efb3c49 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:36.320487 2486 consensus_queue.cc:1035] T a83bae91c29b463aa78412ce2e152ade P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20250114 02:54:36.327194 959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45301f8823ed4ec494e792d41f14f71f" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:36.369807 957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b8361cd538046e0bcb4aac821748f67" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:36.372891 957 raft_consensus.cc:2463] T 0b8361cd538046e0bcb4aac821748f67 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 0.
I20250114 02:54:36.374725 1040 leader_election.cc:304] T 0b8361cd538046e0bcb4aac821748f67 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:36.374910 2449 leader_election.cc:290] T 68981913851243a7a1875fc99efb3c49 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Requested vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:36.375655 2496 raft_consensus.cc:2798] T 0b8361cd538046e0bcb4aac821748f67 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:36.375946 2496 raft_consensus.cc:491] T 0b8361cd538046e0bcb4aac821748f67 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:36.376237 2496 raft_consensus.cc:3054] T 0b8361cd538046e0bcb4aac821748f67 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:36.381742 1103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68981913851243a7a1875fc99efb3c49" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:36.382274 1103 raft_consensus.cc:3054] T 68981913851243a7a1875fc99efb3c49 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:36.387630 1103 raft_consensus.cc:2463] T 68981913851243a7a1875fc99efb3c49 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:36.388598 898 leader_election.cc:304] T 68981913851243a7a1875fc99efb3c49 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:36.390061 2500 raft_consensus.cc:2798] T 68981913851243a7a1875fc99efb3c49 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:36.390506 2500 raft_consensus.cc:695] T 68981913851243a7a1875fc99efb3c49 P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Becoming Leader. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Running, Role: LEADER
I20250114 02:54:36.391467 1205 raft_consensus.cc:383] T 612987f3e4ea4a02932ceefa0d740425 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:36.391743 1205 raft_consensus.cc:738] T 612987f3e4ea4a02932ceefa0d740425 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:36.391290 2500 consensus_queue.cc:237] T 68981913851243a7a1875fc99efb3c49 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:36.392283 1205 consensus_queue.cc:260] T 612987f3e4ea4a02932ceefa0d740425 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:36.359035 481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "87b3db09ac87436e9a60d6de5fedaff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 406 } } ignore_live_leader: false dest_uuid: "b303b7df086340a0b5ad6373940ae750" is_pre_election: true
I20250114 02:54:36.403453 2209 leader_election.cc:304] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87b3db09ac87436e9a60d6de5fedaff5; no voters: 687d4ff6d3cb42cf90a965070b0d8a70, b303b7df086340a0b5ad6373940ae750
I20250114 02:54:36.405755 2422 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:54:36.413187 429 master-stress-test.cc:579] Killed test thread 5/10
I20250114 02:54:36.413480 429 master-stress-test.cc:577] Killing test thread 6/10
I20250114 02:54:36.414398 2447 raft_consensus.cc:491] T a2d68eccaaa64511a9ae36f2b07a7b59 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:36.414800 2447 raft_consensus.cc:513] T a2d68eccaaa64511a9ae36f2b07a7b59 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:36.416674 2447 leader_election.cc:290] T a2d68eccaaa64511a9ae36f2b07a7b59 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:36.418345 1099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2d68eccaaa64511a9ae36f2b07a7b59" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:36.418922 1099 raft_consensus.cc:2463] T a2d68eccaaa64511a9ae36f2b07a7b59 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:36.420310 898 leader_election.cc:304] T a2d68eccaaa64511a9ae36f2b07a7b59 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:36.424423 2500 raft_consensus.cc:2798] T a2d68eccaaa64511a9ae36f2b07a7b59 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:36.424823 2500 raft_consensus.cc:491] T a2d68eccaaa64511a9ae36f2b07a7b59 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:36.425153 2500 raft_consensus.cc:3054] T a2d68eccaaa64511a9ae36f2b07a7b59 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:36.429960 2368 catalog_manager.cc:1260] Loaded cluster ID: f21d09cba41846a78a5fc0502d19012e
I20250114 02:54:36.430253 2368 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5: loading cluster ID for follower catalog manager: success
I20250114 02:54:36.394275 1205 ts_tablet_manager.cc:1428] T 612987f3e4ea4a02932ceefa0d740425 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.086s user 0.004s sys 0.002s
I20250114 02:54:36.445228 1205 tablet_bootstrap.cc:492] T 670dc7ff15d644e18c296eeb2a6b99e1 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:36.448130 2406 raft_consensus.cc:491] T 99b940d872ce4781907517c40fe08781 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 819f9e3412e54d44abb11d8b3ca877c2)
I20250114 02:54:36.448652 2406 raft_consensus.cc:513] T 99b940d872ce4781907517c40fe08781 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:36.450551 2406 leader_election.cc:290] T 99b940d872ce4781907517c40fe08781 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
W20250114 02:54:36.452859 1040 leader_election.cc:343] T 99b940d872ce4781907517c40fe08781 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Not found: Tablet not found: 99b940d872ce4781907517c40fe08781
I20250114 02:54:36.381920 811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68981913851243a7a1875fc99efb3c49" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
W20250114 02:54:36.382517 753 leader_election.cc:343] T 45301f8823ed4ec494e792d41f14f71f P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:36.458209 811 raft_consensus.cc:3054] T 68981913851243a7a1875fc99efb3c49 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
W20250114 02:54:36.458251 753 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.67:45749, user_credentials={real_user=slave}} blocked reactor thread for 75762.8us
I20250114 02:54:36.388525 817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b8361cd538046e0bcb4aac821748f67" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:36.460477 817 raft_consensus.cc:2463] T 0b8361cd538046e0bcb4aac821748f67 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 0.
W20250114 02:54:36.461012 751 leader_election.cc:343] T 45301f8823ed4ec494e792d41f14f71f P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:36.461954 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2d68eccaaa64511a9ae36f2b07a7b59" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:36.462175 751 leader_election.cc:304] T 45301f8823ed4ec494e792d41f14f71f P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:36.462483 818 raft_consensus.cc:2463] T a2d68eccaaa64511a9ae36f2b07a7b59 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
W20250114 02:54:36.462913 751 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.66:44037, user_credentials={real_user=slave}} blocked reactor thread for 77897.3us
I20250114 02:54:36.464687 2372 raft_consensus.cc:2743] T 45301f8823ed4ec494e792d41f14f71f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:36.466271 1205 tablet_bootstrap.cc:654] T 670dc7ff15d644e18c296eeb2a6b99e1 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:36.466776 811 raft_consensus.cc:2463] T 68981913851243a7a1875fc99efb3c49 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
W20250114 02:54:36.472144 1042 leader_election.cc:343] T 99b940d872ce4781907517c40fe08781 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Not found: Tablet not found: 99b940d872ce4781907517c40fe08781
I20250114 02:54:36.472548 1042 leader_election.cc:304] T 99b940d872ce4781907517c40fe08781 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90d6c2c58bb54e0eafb365625ac33871; no voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2
I20250114 02:54:36.473804 2406 raft_consensus.cc:2743] T 99b940d872ce4781907517c40fe08781 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:54:36.476534 2496 raft_consensus.cc:513] T 0b8361cd538046e0bcb4aac821748f67 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:36.448915 2500 raft_consensus.cc:513] T a2d68eccaaa64511a9ae36f2b07a7b59 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:36.479038 957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b8361cd538046e0bcb4aac821748f67" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
I20250114 02:54:36.479468 2500 leader_election.cc:290] T a2d68eccaaa64511a9ae36f2b07a7b59 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Requested vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:36.479633 957 raft_consensus.cc:3054] T 0b8361cd538046e0bcb4aac821748f67 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:36.485101 811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2d68eccaaa64511a9ae36f2b07a7b59" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:36.485845 811 raft_consensus.cc:3054] T a2d68eccaaa64511a9ae36f2b07a7b59 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:36.497808 811 raft_consensus.cc:2463] T a2d68eccaaa64511a9ae36f2b07a7b59 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
W20250114 02:54:36.518112 1042 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.65:39397, user_credentials={real_user=slave}} blocked reactor thread for 44628.8us
I20250114 02:54:36.523643 2406 consensus_queue.cc:1035] T 0b8989d340fa4ae992ef97e226d41110 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:36.526482 1103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2d68eccaaa64511a9ae36f2b07a7b59" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:36.527001 1103 raft_consensus.cc:3054] T a2d68eccaaa64511a9ae36f2b07a7b59 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:36.533021 1103 raft_consensus.cc:2463] T a2d68eccaaa64511a9ae36f2b07a7b59 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:36.536618 898 leader_election.cc:304] T a2d68eccaaa64511a9ae36f2b07a7b59 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:36.538125 2449 raft_consensus.cc:2798] T a2d68eccaaa64511a9ae36f2b07a7b59 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:36.538518 2449 raft_consensus.cc:695] T a2d68eccaaa64511a9ae36f2b07a7b59 P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Becoming Leader. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Running, Role: LEADER
I20250114 02:54:36.539275 2449 consensus_queue.cc:237] T a2d68eccaaa64511a9ae36f2b07a7b59 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:36.558960 2500 raft_consensus.cc:491] T 612987f3e4ea4a02932ceefa0d740425 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:36.559434 2500 raft_consensus.cc:513] T 612987f3e4ea4a02932ceefa0d740425 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:36.561427 2500 leader_election.cc:290] T 612987f3e4ea4a02932ceefa0d740425 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:36.500411 1206 tablet_bootstrap.cc:492] T e7ca0bd1dbd542de9ee11e6c72522ae0 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:36.563064 815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "612987f3e4ea4a02932ceefa0d740425" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:36.565626 815 raft_consensus.cc:2463] T 612987f3e4ea4a02932ceefa0d740425 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:36.565639 1206 ts_tablet_manager.cc:1397] T e7ca0bd1dbd542de9ee11e6c72522ae0 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.230s user 0.009s sys 0.009s
I20250114 02:54:36.566918 898 leader_election.cc:304] T 612987f3e4ea4a02932ceefa0d740425 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
I20250114 02:54:36.568049 1206 raft_consensus.cc:357] T e7ca0bd1dbd542de9ee11e6c72522ae0 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:36.568722 1206 raft_consensus.cc:383] T e7ca0bd1dbd542de9ee11e6c72522ae0 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:36.568958 1206 raft_consensus.cc:738] T e7ca0bd1dbd542de9ee11e6c72522ae0 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:36.568619 2500 raft_consensus.cc:2798] T 612987f3e4ea4a02932ceefa0d740425 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:36.569594 2500 raft_consensus.cc:491] T 612987f3e4ea4a02932ceefa0d740425 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:36.569970 2500 raft_consensus.cc:3054] T 612987f3e4ea4a02932ceefa0d740425 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:36.569504 1206 consensus_queue.cc:260] T e7ca0bd1dbd542de9ee11e6c72522ae0 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:36.576531 1206 ts_tablet_manager.cc:1428] T e7ca0bd1dbd542de9ee11e6c72522ae0 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20250114 02:54:36.577442 1206 tablet_bootstrap.cc:492] T 038ea1ea231e490ba27ecd5430883058 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:36.586064 957 raft_consensus.cc:2463] T 0b8361cd538046e0bcb4aac821748f67 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1.
I20250114 02:54:36.587469 2496 leader_election.cc:290] T 0b8361cd538046e0bcb4aac821748f67 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Requested vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:36.591620 1207 tablet_bootstrap.cc:492] T d56255ca806d4e2cac09fbb5e1b1efbd P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:36.593950 1205 tablet_bootstrap.cc:492] T 670dc7ff15d644e18c296eeb2a6b99e1 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:36.595834 1205 ts_tablet_manager.cc:1397] T 670dc7ff15d644e18c296eeb2a6b99e1 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.151s user 0.005s sys 0.009s
I20250114 02:54:36.598816 1205 raft_consensus.cc:357] T 670dc7ff15d644e18c296eeb2a6b99e1 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:36.599426 1206 tablet_bootstrap.cc:654] T 038ea1ea231e490ba27ecd5430883058 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:36.588768 2500 raft_consensus.cc:513] T 612987f3e4ea4a02932ceefa0d740425 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:36.603222 811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "612987f3e4ea4a02932ceefa0d740425" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:36.604029 811 raft_consensus.cc:3054] T 612987f3e4ea4a02932ceefa0d740425 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:36.604887 2500 leader_election.cc:290] T 612987f3e4ea4a02932ceefa0d740425 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Requested vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:36.605862 1207 tablet_bootstrap.cc:654] T d56255ca806d4e2cac09fbb5e1b1efbd P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:36.600044 1205 raft_consensus.cc:383] T 670dc7ff15d644e18c296eeb2a6b99e1 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:36.606393 1205 raft_consensus.cc:738] T 670dc7ff15d644e18c296eeb2a6b99e1 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:36.607203 1205 consensus_queue.cc:260] T 670dc7ff15d644e18c296eeb2a6b99e1 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:36.612162 813 raft_consensus.cc:2463] T d56255ca806d4e2cac09fbb5e1b1efbd P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:36.613183 898 leader_election.cc:304] T d56255ca806d4e2cac09fbb5e1b1efbd P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters: 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:36.614157 2500 raft_consensus.cc:2798] T d56255ca806d4e2cac09fbb5e1b1efbd P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:36.614692 2500 raft_consensus.cc:695] T d56255ca806d4e2cac09fbb5e1b1efbd P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Becoming Leader. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Running, Role: LEADER
I20250114 02:54:36.615605 2500 consensus_queue.cc:237] T d56255ca806d4e2cac09fbb5e1b1efbd P 525198593c8e47afaee54e74267b87e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:36.616940 1205 ts_tablet_manager.cc:1428] T 670dc7ff15d644e18c296eeb2a6b99e1 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.021s user 0.006s sys 0.000s
I20250114 02:54:36.617866 1205 tablet_bootstrap.cc:492] T be7aad41ccdd4ef9bcfe8a985155708f P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:36.625308 811 raft_consensus.cc:2463] T 612987f3e4ea4a02932ceefa0d740425 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:36.626510 898 leader_election.cc:304] T 612987f3e4ea4a02932ceefa0d740425 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
I20250114 02:54:36.627676 2504 raft_consensus.cc:2798] T 612987f3e4ea4a02932ceefa0d740425 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:36.628145 2504 raft_consensus.cc:695] T 612987f3e4ea4a02932ceefa0d740425 P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Becoming Leader. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Running, Role: LEADER
I20250114 02:54:36.628989 2504 consensus_queue.cc:237] T 612987f3e4ea4a02932ceefa0d740425 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:36.637964 1205 tablet_bootstrap.cc:654] T be7aad41ccdd4ef9bcfe8a985155708f P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:36.642359 1207 tablet_bootstrap.cc:492] T d56255ca806d4e2cac09fbb5e1b1efbd P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:36.642771 1207 ts_tablet_manager.cc:1397] T d56255ca806d4e2cac09fbb5e1b1efbd P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.051s user 0.008s sys 0.001s
I20250114 02:54:36.645609 1207 raft_consensus.cc:357] T d56255ca806d4e2cac09fbb5e1b1efbd P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:36.646569 1207 raft_consensus.cc:383] T d56255ca806d4e2cac09fbb5e1b1efbd P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:36.646903 1207 raft_consensus.cc:738] T d56255ca806d4e2cac09fbb5e1b1efbd P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:36.647644 1207 consensus_queue.cc:260] T d56255ca806d4e2cac09fbb5e1b1efbd P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:36.661707 1207 ts_tablet_manager.cc:1428] T d56255ca806d4e2cac09fbb5e1b1efbd P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.019s user 0.008s sys 0.000s
I20250114 02:54:36.662633 1207 tablet_bootstrap.cc:492] T 2e17f619ed2745078541cc1c924aff96 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:36.667307 2447 raft_consensus.cc:491] T 670dc7ff15d644e18c296eeb2a6b99e1 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:36.667824 2447 raft_consensus.cc:513] T 670dc7ff15d644e18c296eeb2a6b99e1 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:36.670456 811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "670dc7ff15d644e18c296eeb2a6b99e1" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:36.671478 811 raft_consensus.cc:2388] T 670dc7ff15d644e18c296eeb2a6b99e1 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 525198593c8e47afaee54e74267b87e3 in current term 1: Already voted for candidate 90d6c2c58bb54e0eafb365625ac33871 in this term.
I20250114 02:54:36.671734 1207 tablet_bootstrap.cc:654] T 2e17f619ed2745078541cc1c924aff96 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:36.677850 1205 tablet_bootstrap.cc:492] T be7aad41ccdd4ef9bcfe8a985155708f P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:36.678270 1205 ts_tablet_manager.cc:1397] T be7aad41ccdd4ef9bcfe8a985155708f P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.061s user 0.013s sys 0.000s
I20250114 02:54:36.685998 1205 raft_consensus.cc:357] T be7aad41ccdd4ef9bcfe8a985155708f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:36.686703 1205 raft_consensus.cc:383] T be7aad41ccdd4ef9bcfe8a985155708f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:36.686980 1205 raft_consensus.cc:738] T be7aad41ccdd4ef9bcfe8a985155708f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:36.695642 2447 leader_election.cc:290] T 670dc7ff15d644e18c296eeb2a6b99e1 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:36.716490 1205 consensus_queue.cc:260] T be7aad41ccdd4ef9bcfe8a985155708f P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:36.720523 1205 ts_tablet_manager.cc:1428] T be7aad41ccdd4ef9bcfe8a985155708f P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.042s user 0.007s sys 0.001s
I20250114 02:54:36.721439 1205 tablet_bootstrap.cc:492] T 797d6cdf36704c98aa106aac87a945ca P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:36.723765 1206 tablet_bootstrap.cc:492] T 038ea1ea231e490ba27ecd5430883058 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:36.724226 1206 ts_tablet_manager.cc:1397] T 038ea1ea231e490ba27ecd5430883058 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.147s user 0.009s sys 0.000s
I20250114 02:54:36.726495 1206 raft_consensus.cc:357] T 038ea1ea231e490ba27ecd5430883058 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:36.727231 1206 raft_consensus.cc:383] T 038ea1ea231e490ba27ecd5430883058 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:36.727532 1206 raft_consensus.cc:738] T 038ea1ea231e490ba27ecd5430883058 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:36.727892 1205 tablet_bootstrap.cc:654] T 797d6cdf36704c98aa106aac87a945ca P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:36.728297 1206 consensus_queue.cc:260] T 038ea1ea231e490ba27ecd5430883058 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:36.740348 2368 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5: acquiring CA information for follower catalog manager: success
I20250114 02:54:36.747437 1205 tablet_bootstrap.cc:492] T 797d6cdf36704c98aa106aac87a945ca P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:36.747859 1205 ts_tablet_manager.cc:1397] T 797d6cdf36704c98aa106aac87a945ca P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.027s user 0.007s sys 0.003s
I20250114 02:54:36.752991 1205 raft_consensus.cc:357] T 797d6cdf36704c98aa106aac87a945ca P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:36.753695 1205 raft_consensus.cc:383] T 797d6cdf36704c98aa106aac87a945ca P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:36.753940 1205 raft_consensus.cc:738] T 797d6cdf36704c98aa106aac87a945ca P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:36.780915 1205 consensus_queue.cc:260] T 797d6cdf36704c98aa106aac87a945ca P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:36.783643 1205 ts_tablet_manager.cc:1428] T 797d6cdf36704c98aa106aac87a945ca P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.036s user 0.004s sys 0.003s
I20250114 02:54:36.784595 1205 tablet_bootstrap.cc:492] T 2e17f619ed2745078541cc1c924aff96 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:36.790822 1205 tablet_bootstrap.cc:654] T 2e17f619ed2745078541cc1c924aff96 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:36.790735 1040 leader_election.cc:304] T 0b8361cd538046e0bcb4aac821748f67 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:36.792301 813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b8361cd538046e0bcb4aac821748f67" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:36.792888 813 raft_consensus.cc:3054] T 0b8361cd538046e0bcb4aac821748f67 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:36.798729 2510 raft_consensus.cc:2798] T 0b8361cd538046e0bcb4aac821748f67 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:36.799986 2510 raft_consensus.cc:695] T 0b8361cd538046e0bcb4aac821748f67 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Becoming Leader. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Running, Role: LEADER
I20250114 02:54:36.801666 2510 consensus_queue.cc:237] T 0b8361cd538046e0bcb4aac821748f67 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:36.803830 1205 tablet_bootstrap.cc:492] T 2e17f619ed2745078541cc1c924aff96 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:36.804344 1205 ts_tablet_manager.cc:1397] T 2e17f619ed2745078541cc1c924aff96 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.007s
I20250114 02:54:36.806968 1205 raft_consensus.cc:357] T 2e17f619ed2745078541cc1c924aff96 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:36.807593 1207 tablet_bootstrap.cc:492] T 2e17f619ed2745078541cc1c924aff96 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:36.807662 1205 raft_consensus.cc:383] T 2e17f619ed2745078541cc1c924aff96 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:36.807896 1205 raft_consensus.cc:738] T 2e17f619ed2745078541cc1c924aff96 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:36.808468 1207 ts_tablet_manager.cc:1397] T 2e17f619ed2745078541cc1c924aff96 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.146s user 0.006s sys 0.006s
I20250114 02:54:36.808686 1205 consensus_queue.cc:260] T 2e17f619ed2745078541cc1c924aff96 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:36.812615 1207 raft_consensus.cc:357] T 2e17f619ed2745078541cc1c924aff96 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:36.813572 1207 raft_consensus.cc:383] T 2e17f619ed2745078541cc1c924aff96 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:36.813900 1207 raft_consensus.cc:738] T 2e17f619ed2745078541cc1c924aff96 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:36.814786 1207 consensus_queue.cc:260] T 2e17f619ed2745078541cc1c924aff96 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:36.817207 1207 ts_tablet_manager.cc:1428] T 2e17f619ed2745078541cc1c924aff96 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.008s user 0.007s sys 0.001s
I20250114 02:54:36.821138 1205 ts_tablet_manager.cc:1428] T 2e17f619ed2745078541cc1c924aff96 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
I20250114 02:54:36.821971 1207 tablet_bootstrap.cc:492] T e06daadd346d45969ad5efcf03bc0870 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:36.822024 1205 tablet_bootstrap.cc:492] T 038ea1ea231e490ba27ecd5430883058 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:36.833606 1205 tablet_bootstrap.cc:654] T 038ea1ea231e490ba27ecd5430883058 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:36.834597 2372 raft_consensus.cc:491] T 038ea1ea231e490ba27ecd5430883058 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:36.835026 2372 raft_consensus.cc:513] T 038ea1ea231e490ba27ecd5430883058 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:36.836946 2372 leader_election.cc:290] T 038ea1ea231e490ba27ecd5430883058 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:36.838717 960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "038ea1ea231e490ba27ecd5430883058" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
W20250114 02:54:36.841514 751 leader_election.cc:343] T 038ea1ea231e490ba27ecd5430883058 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:36.843683 813 raft_consensus.cc:2463] T 0b8361cd538046e0bcb4aac821748f67 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1.
I20250114 02:54:36.847275 1205 tablet_bootstrap.cc:492] T 038ea1ea231e490ba27ecd5430883058 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:36.847694 1205 ts_tablet_manager.cc:1397] T 038ea1ea231e490ba27ecd5430883058 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.026s user 0.014s sys 0.001s
I20250114 02:54:36.730624 1206 ts_tablet_manager.cc:1428] T 038ea1ea231e490ba27ecd5430883058 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250114 02:54:36.850193 1205 raft_consensus.cc:357] T 038ea1ea231e490ba27ecd5430883058 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:36.850850 1205 raft_consensus.cc:383] T 038ea1ea231e490ba27ecd5430883058 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:36.851073 1205 raft_consensus.cc:738] T 038ea1ea231e490ba27ecd5430883058 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:36.851578 1205 consensus_queue.cc:260] T 038ea1ea231e490ba27ecd5430883058 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:36.853915 1205 ts_tablet_manager.cc:1428] T 038ea1ea231e490ba27ecd5430883058 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20250114 02:54:36.854817 1205 tablet_bootstrap.cc:492] T ed109b49f20642f988161a3df7872419 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:36.868449 2372 raft_consensus.cc:491] T 2e17f619ed2745078541cc1c924aff96 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:36.868943 2372 raft_consensus.cc:513] T 2e17f619ed2745078541cc1c924aff96 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:36.870976 1205 tablet_bootstrap.cc:654] T ed109b49f20642f988161a3df7872419 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:36.871243 2372 leader_election.cc:290] T 2e17f619ed2745078541cc1c924aff96 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:36.873412 960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e17f619ed2745078541cc1c924aff96" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:36.874064 960 raft_consensus.cc:2463] T 2e17f619ed2745078541cc1c924aff96 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 0.
I20250114 02:54:36.875411 751 leader_election.cc:304] T 2e17f619ed2745078541cc1c924aff96 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
I20250114 02:54:36.876044 1205 tablet_bootstrap.cc:492] T ed109b49f20642f988161a3df7872419 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:36.876480 1205 ts_tablet_manager.cc:1397] T ed109b49f20642f988161a3df7872419 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.000s
I20250114 02:54:36.876443 2372 raft_consensus.cc:2798] T 2e17f619ed2745078541cc1c924aff96 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:36.876885 2372 raft_consensus.cc:491] T 2e17f619ed2745078541cc1c924aff96 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:36.877238 2372 raft_consensus.cc:3054] T 2e17f619ed2745078541cc1c924aff96 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:36.878835 1205 raft_consensus.cc:357] T ed109b49f20642f988161a3df7872419 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:36.879520 1205 raft_consensus.cc:383] T ed109b49f20642f988161a3df7872419 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:36.879781 1205 raft_consensus.cc:738] T ed109b49f20642f988161a3df7872419 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:36.880494 1205 consensus_queue.cc:260] T ed109b49f20642f988161a3df7872419 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:36.882984 1205 ts_tablet_manager.cc:1428] T ed109b49f20642f988161a3df7872419 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250114 02:54:36.883821 1205 tablet_bootstrap.cc:492] T 97de018391f84b0ebd3130861681b7b5 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:36.885089 2372 raft_consensus.cc:513] T 2e17f619ed2745078541cc1c924aff96 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:36.888939 1205 tablet_bootstrap.cc:654] T 97de018391f84b0ebd3130861681b7b5 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:36.891815 960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e17f619ed2745078541cc1c924aff96" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
I20250114 02:54:36.892331 960 raft_consensus.cc:3054] T 2e17f619ed2745078541cc1c924aff96 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:36.898651 960 raft_consensus.cc:2463] T 2e17f619ed2745078541cc1c924aff96 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 1.
I20250114 02:54:36.899865 751 leader_election.cc:304] T 2e17f619ed2745078541cc1c924aff96 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
I20250114 02:54:36.888422 1206 tablet_bootstrap.cc:492] T 6d741bfd00784d2d830be3ebe50dc7c4 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:36.900719 2484 raft_consensus.cc:2798] T 2e17f619ed2745078541cc1c924aff96 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election won for term 1
W20250114 02:54:36.907640 1042 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.65:39397, user_credentials={real_user=slave}} blocked reactor thread for 116405us
I20250114 02:54:36.912984 2484 raft_consensus.cc:695] T 2e17f619ed2745078541cc1c924aff96 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 LEADER]: Becoming Leader. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Running, Role: LEADER
I20250114 02:54:36.914148 2484 consensus_queue.cc:237] T 2e17f619ed2745078541cc1c924aff96 P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:36.918139 2372 leader_election.cc:290] T 2e17f619ed2745078541cc1c924aff96 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Requested vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:36.920675 1103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "612987f3e4ea4a02932ceefa0d740425" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:36.921599 1103 raft_consensus.cc:2463] T 612987f3e4ea4a02932ceefa0d740425 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
W20250114 02:54:36.932350 457 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet d2a7b2bee5c1440fb705b172c18a9247 (table default.Table_3c0c8ba6eec34be4a6ea569d66038106 [id=4d2f7fabf8a44155a586a2b79a8fca53], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet d2a7b2bee5c1440fb705b172c18a9247 consensus state has no leader
I20250114 02:54:36.934657 1207 tablet_bootstrap.cc:654] T e06daadd346d45969ad5efcf03bc0870 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
W20250114 02:54:36.948652 1040 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.66:44037, user_credentials={real_user=slave}} blocked reactor thread for 158345us
I20250114 02:54:36.964627 1103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "612987f3e4ea4a02932ceefa0d740425" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:36.965431 1103 raft_consensus.cc:3054] T 612987f3e4ea4a02932ceefa0d740425 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:36.966938 2534 raft_consensus.cc:491] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b303b7df086340a0b5ad6373940ae750)
I20250114 02:54:36.967404 2534 raft_consensus.cc:513] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b303b7df086340a0b5ad6373940ae750" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 45367 } } peers { permanent_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } } peers { permanent_uuid: "87b3db09ac87436e9a60d6de5fedaff5" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } }
I20250114 02:54:36.982821 2500 consensus_queue.cc:1035] T 5d545ac27a7f49faace0655713405cea P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:36.986987 2368 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250114 02:54:36.993852 1103 raft_consensus.cc:2463] T 612987f3e4ea4a02932ceefa0d740425 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:37.002882 481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "87b3db09ac87436e9a60d6de5fedaff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 406 } } ignore_live_leader: false dest_uuid: "b303b7df086340a0b5ad6373940ae750" is_pre_election: true
I20250114 02:54:37.012854 555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "87b3db09ac87436e9a60d6de5fedaff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 406 } } ignore_live_leader: false dest_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" is_pre_election: true
I20250114 02:54:37.014612 2204 leader_election.cc:304] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87b3db09ac87436e9a60d6de5fedaff5; no voters: 687d4ff6d3cb42cf90a965070b0d8a70, b303b7df086340a0b5ad6373940ae750
I20250114 02:54:37.021713 1206 tablet_bootstrap.cc:654] T 6d741bfd00784d2d830be3ebe50dc7c4 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:37.027756 2534 leader_election.cc:290] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b303b7df086340a0b5ad6373940ae750 (127.0.107.126:45367), 687d4ff6d3cb42cf90a965070b0d8a70 (127.0.107.125:46837)
I20250114 02:54:37.028393 2534 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:54:37.053795 1205 tablet_bootstrap.cc:492] T 97de018391f84b0ebd3130861681b7b5 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:37.054191 1205 ts_tablet_manager.cc:1397] T 97de018391f84b0ebd3130861681b7b5 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.171s user 0.033s sys 0.041s
I20250114 02:54:37.000954 1099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "670dc7ff15d644e18c296eeb2a6b99e1" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:37.061468 1205 raft_consensus.cc:357] T 97de018391f84b0ebd3130861681b7b5 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:37.062121 1205 raft_consensus.cc:383] T 97de018391f84b0ebd3130861681b7b5 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:37.062387 1205 raft_consensus.cc:738] T 97de018391f84b0ebd3130861681b7b5 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:37.062940 1205 consensus_queue.cc:260] T 97de018391f84b0ebd3130861681b7b5 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:37.063861 2389 raft_consensus.cc:491] T be7aad41ccdd4ef9bcfe8a985155708f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:37.064343 2389 raft_consensus.cc:513] T be7aad41ccdd4ef9bcfe8a985155708f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:37.065500 1207 tablet_bootstrap.cc:492] T e06daadd346d45969ad5efcf03bc0870 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:37.065910 1207 ts_tablet_manager.cc:1397] T e06daadd346d45969ad5efcf03bc0870 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.244s user 0.015s sys 0.017s
I20250114 02:54:37.066183 2389 leader_election.cc:290] T be7aad41ccdd4ef9bcfe8a985155708f P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:37.074111 1099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "038ea1ea231e490ba27ecd5430883058" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:37.079506 1099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e17f619ed2745078541cc1c924aff96" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:37.080062 1099 raft_consensus.cc:2463] T 2e17f619ed2745078541cc1c924aff96 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 0.
I20250114 02:54:37.087886 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be7aad41ccdd4ef9bcfe8a985155708f" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:37.088183 1205 ts_tablet_manager.cc:1428] T 97de018391f84b0ebd3130861681b7b5 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.034s user 0.007s sys 0.000s
I20250114 02:54:37.088513 818 raft_consensus.cc:2463] T be7aad41ccdd4ef9bcfe8a985155708f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 0.
I20250114 02:54:37.086776 1207 raft_consensus.cc:357] T e06daadd346d45969ad5efcf03bc0870 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:37.089015 1207 raft_consensus.cc:383] T e06daadd346d45969ad5efcf03bc0870 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:37.089258 1207 raft_consensus.cc:738] T e06daadd346d45969ad5efcf03bc0870 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:37.089813 1207 consensus_queue.cc:260] T e06daadd346d45969ad5efcf03bc0870 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:37.091385 1207 ts_tablet_manager.cc:1428] T e06daadd346d45969ad5efcf03bc0870 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.025s user 0.003s sys 0.001s
I20250114 02:54:37.091265 1099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e17f619ed2745078541cc1c924aff96" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:37.091895 1099 raft_consensus.cc:3054] T 2e17f619ed2745078541cc1c924aff96 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:37.092303 1207 tablet_bootstrap.cc:492] T ed109b49f20642f988161a3df7872419 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
W20250114 02:54:37.097522 898 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.67:45749, user_credentials={real_user=slave}} blocked reactor thread for 50551.2us
I20250114 02:54:37.100303 1099 raft_consensus.cc:2463] T 2e17f619ed2745078541cc1c924aff96 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 1.
I20250114 02:54:37.103762 960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be7aad41ccdd4ef9bcfe8a985155708f" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:37.104362 960 raft_consensus.cc:2463] T be7aad41ccdd4ef9bcfe8a985155708f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 0.
I20250114 02:54:37.108879 2372 raft_consensus.cc:491] T 97de018391f84b0ebd3130861681b7b5 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:37.109293 2372 raft_consensus.cc:513] T 97de018391f84b0ebd3130861681b7b5 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:37.111179 2372 leader_election.cc:290] T 97de018391f84b0ebd3130861681b7b5 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
W20250114 02:54:37.112136 753 leader_election.cc:343] T 038ea1ea231e490ba27ecd5430883058 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:37.112536 753 leader_election.cc:304] T 038ea1ea231e490ba27ecd5430883058 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:37.113014 1205 tablet_bootstrap.cc:492] T e06daadd346d45969ad5efcf03bc0870 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:37.113133 2484 raft_consensus.cc:2743] T 038ea1ea231e490ba27ecd5430883058 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:37.108604 1042 leader_election.cc:304] T be7aad41ccdd4ef9bcfe8a985155708f P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:37.117327 1207 tablet_bootstrap.cc:654] T ed109b49f20642f988161a3df7872419 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:37.118253 1205 tablet_bootstrap.cc:654] T e06daadd346d45969ad5efcf03bc0870 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:37.119148 2389 raft_consensus.cc:2798] T be7aad41ccdd4ef9bcfe8a985155708f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:37.119709 2389 raft_consensus.cc:491] T be7aad41ccdd4ef9bcfe8a985155708f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:37.120092 2449 raft_consensus.cc:491] T 797d6cdf36704c98aa106aac87a945ca P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:37.120092 2389 raft_consensus.cc:3054] T be7aad41ccdd4ef9bcfe8a985155708f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:37.120543 2449 raft_consensus.cc:513] T 797d6cdf36704c98aa106aac87a945ca P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:37.124058 2449 leader_election.cc:290] T 797d6cdf36704c98aa106aac87a945ca P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:37.127236 2389 raft_consensus.cc:513] T be7aad41ccdd4ef9bcfe8a985155708f P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
W20250114 02:54:37.130610 2204 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.125:46837, user_credentials={real_user=slave}} blocked reactor thread for 116204us
I20250114 02:54:37.134654 1207 tablet_bootstrap.cc:492] T ed109b49f20642f988161a3df7872419 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:37.145509 1207 ts_tablet_manager.cc:1397] T ed109b49f20642f988161a3df7872419 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.053s user 0.012s sys 0.001s
I20250114 02:54:37.146472 2389 leader_election.cc:290] T be7aad41ccdd4ef9bcfe8a985155708f P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Requested vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:37.151479 815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be7aad41ccdd4ef9bcfe8a985155708f" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:37.152143 815 raft_consensus.cc:3054] T be7aad41ccdd4ef9bcfe8a985155708f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:37.100668 898 leader_election.cc:304] T 670dc7ff15d644e18c296eeb2a6b99e1 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 525198593c8e47afaee54e74267b87e3; no voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:37.159021 815 raft_consensus.cc:2463] T be7aad41ccdd4ef9bcfe8a985155708f P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1.
I20250114 02:54:37.160614 958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be7aad41ccdd4ef9bcfe8a985155708f" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
I20250114 02:54:37.147831 1207 raft_consensus.cc:357] T ed109b49f20642f988161a3df7872419 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:37.161365 958 raft_consensus.cc:3054] T be7aad41ccdd4ef9bcfe8a985155708f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:37.161572 1207 raft_consensus.cc:383] T ed109b49f20642f988161a3df7872419 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:37.161855 1207 raft_consensus.cc:738] T ed109b49f20642f988161a3df7872419 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:37.162436 1207 consensus_queue.cc:260] T ed109b49f20642f988161a3df7872419 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:37.165719 1207 ts_tablet_manager.cc:1428] T ed109b49f20642f988161a3df7872419 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.020s user 0.007s sys 0.000s
I20250114 02:54:37.166700 1207 tablet_bootstrap.cc:492] T 97de018391f84b0ebd3130861681b7b5 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:37.167332 1042 leader_election.cc:304] T be7aad41ccdd4ef9bcfe8a985155708f P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:37.168047 2389 raft_consensus.cc:2798] T be7aad41ccdd4ef9bcfe8a985155708f P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:37.168758 2389 raft_consensus.cc:695] T be7aad41ccdd4ef9bcfe8a985155708f P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Becoming Leader. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Running, Role: LEADER
I20250114 02:54:37.169607 2389 consensus_queue.cc:237] T be7aad41ccdd4ef9bcfe8a985155708f P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:37.161039 1099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97de018391f84b0ebd3130861681b7b5" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
W20250114 02:54:37.172960 753 leader_election.cc:343] T 97de018391f84b0ebd3130861681b7b5 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:37.177769 958 raft_consensus.cc:2463] T be7aad41ccdd4ef9bcfe8a985155708f P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1.
I20250114 02:54:37.163848 1206 tablet_bootstrap.cc:492] T 6d741bfd00784d2d830be3ebe50dc7c4 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:37.180804 1206 ts_tablet_manager.cc:1397] T 6d741bfd00784d2d830be3ebe50dc7c4 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.293s user 0.009s sys 0.007s
I20250114 02:54:37.172099 1207 tablet_bootstrap.cc:654] T 97de018391f84b0ebd3130861681b7b5 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:37.194725 2500 raft_consensus.cc:3054] T 670dc7ff15d644e18c296eeb2a6b99e1 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
W20250114 02:54:37.200760 898 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.67:45749, user_credentials={real_user=slave}} blocked reactor thread for 100159us
I20250114 02:54:37.205148 1100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "797d6cdf36704c98aa106aac87a945ca" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:37.206634 1205 tablet_bootstrap.cc:492] T e06daadd346d45969ad5efcf03bc0870 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:37.207098 1205 ts_tablet_manager.cc:1397] T e06daadd346d45969ad5efcf03bc0870 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.094s user 0.013s sys 0.000s
I20250114 02:54:37.210734 1205 raft_consensus.cc:357] T e06daadd346d45969ad5efcf03bc0870 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:37.212317 1205 raft_consensus.cc:383] T e06daadd346d45969ad5efcf03bc0870 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:37.212702 1205 raft_consensus.cc:738] T e06daadd346d45969ad5efcf03bc0870 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:37.213526 811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "797d6cdf36704c98aa106aac87a945ca" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:37.213320 1205 consensus_queue.cc:260] T e06daadd346d45969ad5efcf03bc0870 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:37.214085 811 raft_consensus.cc:2463] T 797d6cdf36704c98aa106aac87a945ca P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:37.214799 2500 raft_consensus.cc:2743] T 670dc7ff15d644e18c296eeb2a6b99e1 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:37.221340 2484 raft_consensus.cc:491] T e7ca0bd1dbd542de9ee11e6c72522ae0 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:37.221781 2484 raft_consensus.cc:513] T e7ca0bd1dbd542de9ee11e6c72522ae0 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:37.233565 2484 leader_election.cc:290] T e7ca0bd1dbd542de9ee11e6c72522ae0 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:37.235284 1099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7ca0bd1dbd542de9ee11e6c72522ae0" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
W20250114 02:54:37.236485 753 leader_election.cc:343] T e7ca0bd1dbd542de9ee11e6c72522ae0 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:37.220130 1206 raft_consensus.cc:357] T 6d741bfd00784d2d830be3ebe50dc7c4 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:37.247776 1206 raft_consensus.cc:383] T 6d741bfd00784d2d830be3ebe50dc7c4 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:37.247932 1207 tablet_bootstrap.cc:492] T 97de018391f84b0ebd3130861681b7b5 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:37.248381 1207 ts_tablet_manager.cc:1397] T 97de018391f84b0ebd3130861681b7b5 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.082s user 0.013s sys 0.001s
I20250114 02:54:37.249346 958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97de018391f84b0ebd3130861681b7b5" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:37.250001 958 raft_consensus.cc:2463] T 97de018391f84b0ebd3130861681b7b5 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 0.
I20250114 02:54:37.248031 1206 raft_consensus.cc:738] T 6d741bfd00784d2d830be3ebe50dc7c4 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:37.254513 1206 consensus_queue.cc:260] T 6d741bfd00784d2d830be3ebe50dc7c4 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:37.257408 1205 ts_tablet_manager.cc:1428] T e06daadd346d45969ad5efcf03bc0870 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.050s user 0.003s sys 0.001s
I20250114 02:54:37.258682 2500 consensus_queue.cc:1035] T 5d545ac27a7f49faace0655713405cea P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250114 02:54:37.260861 898 leader_election.cc:343] T 797d6cdf36704c98aa106aac87a945ca P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:37.256471 1206 ts_tablet_manager.cc:1428] T 6d741bfd00784d2d830be3ebe50dc7c4 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.075s user 0.005s sys 0.001s
I20250114 02:54:37.267762 1206 tablet_bootstrap.cc:492] T a2c3f3a89b3d47f1b0dfacfac8218805 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:37.274216 898 leader_election.cc:304] T 797d6cdf36704c98aa106aac87a945ca P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters: 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:37.275035 2500 raft_consensus.cc:2798] T 797d6cdf36704c98aa106aac87a945ca P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:37.275355 2500 raft_consensus.cc:491] T 797d6cdf36704c98aa106aac87a945ca P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:37.275777 2500 raft_consensus.cc:3054] T 797d6cdf36704c98aa106aac87a945ca P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:37.276346 2389 consensus_queue.cc:1035] T 670dc7ff15d644e18c296eeb2a6b99e1 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:37.279176 1207 raft_consensus.cc:357] T 97de018391f84b0ebd3130861681b7b5 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:37.279974 1207 raft_consensus.cc:383] T 97de018391f84b0ebd3130861681b7b5 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:37.280246 1207 raft_consensus.cc:738] T 97de018391f84b0ebd3130861681b7b5 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:37.290136 1207 consensus_queue.cc:260] T 97de018391f84b0ebd3130861681b7b5 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:37.293727 2389 raft_consensus.cc:491] T ed109b49f20642f988161a3df7872419 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:37.294152 2389 raft_consensus.cc:513] T ed109b49f20642f988161a3df7872419 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:37.294775 751 leader_election.cc:304] T 97de018391f84b0ebd3130861681b7b5 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters: 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:37.295791 2484 raft_consensus.cc:2798] T 97de018391f84b0ebd3130861681b7b5 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:37.296175 2484 raft_consensus.cc:491] T 97de018391f84b0ebd3130861681b7b5 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:37.296497 2484 raft_consensus.cc:3054] T 97de018391f84b0ebd3130861681b7b5 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:37.297312 959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed109b49f20642f988161a3df7872419" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:37.297866 959 raft_consensus.cc:2463] T ed109b49f20642f988161a3df7872419 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 0.
I20250114 02:54:37.298822 1040 leader_election.cc:304] T ed109b49f20642f988161a3df7872419 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:37.299634 2528 raft_consensus.cc:2798] T ed109b49f20642f988161a3df7872419 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:37.299916 2528 raft_consensus.cc:491] T ed109b49f20642f988161a3df7872419 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:37.300189 2528 raft_consensus.cc:3054] T ed109b49f20642f988161a3df7872419 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:37.301290 960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7ca0bd1dbd542de9ee11e6c72522ae0" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:37.301275 814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed109b49f20642f988161a3df7872419" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:37.301827 814 raft_consensus.cc:2463] T ed109b49f20642f988161a3df7872419 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 0.
I20250114 02:54:37.303068 2449 raft_consensus.cc:491] T 038ea1ea231e490ba27ecd5430883058 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:37.303483 2449 raft_consensus.cc:513] T 038ea1ea231e490ba27ecd5430883058 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:37.292529 1207 ts_tablet_manager.cc:1428] T 97de018391f84b0ebd3130861681b7b5 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.044s user 0.006s sys 0.000s
I20250114 02:54:37.306128 2484 raft_consensus.cc:513] T 97de018391f84b0ebd3130861681b7b5 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:37.306993 2449 leader_election.cc:290] T 038ea1ea231e490ba27ecd5430883058 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:37.307791 2389 leader_election.cc:290] T ed109b49f20642f988161a3df7872419 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:37.311940 2484 leader_election.cc:290] T 97de018391f84b0ebd3130861681b7b5 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Requested vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:37.313369 1099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97de018391f84b0ebd3130861681b7b5" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:37.313891 1099 raft_consensus.cc:3054] T 97de018391f84b0ebd3130861681b7b5 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:37.331563 960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97de018391f84b0ebd3130861681b7b5" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
I20250114 02:54:37.332116 960 raft_consensus.cc:3054] T 97de018391f84b0ebd3130861681b7b5 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:37.333443 1205 tablet_bootstrap.cc:492] T e7ca0bd1dbd542de9ee11e6c72522ae0 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:37.334483 1104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "038ea1ea231e490ba27ecd5430883058" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:37.335157 2406 raft_consensus.cc:491] T 3161f75bc4e84000af022faa9804c810 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:37.335566 2406 raft_consensus.cc:513] T 3161f75bc4e84000af022faa9804c810 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:37.335765 816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "038ea1ea231e490ba27ecd5430883058" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:37.336342 816 raft_consensus.cc:2463] T 038ea1ea231e490ba27ecd5430883058 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:37.339793 2406 leader_election.cc:290] T 3161f75bc4e84000af022faa9804c810 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
W20250114 02:54:37.345000 1040 leader_election.cc:343] T 3161f75bc4e84000af022faa9804c810 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Not found: Tablet not found: 3161f75bc4e84000af022faa9804c810
I20250114 02:54:37.351492 960 raft_consensus.cc:2463] T 97de018391f84b0ebd3130861681b7b5 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 1.
I20250114 02:54:37.352257 2500 raft_consensus.cc:513] T 797d6cdf36704c98aa106aac87a945ca P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
W20250114 02:54:37.348093 751 leader_election.cc:343] T e7ca0bd1dbd542de9ee11e6c72522ae0 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:37.354149 2500 leader_election.cc:290] T 797d6cdf36704c98aa106aac87a945ca P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Requested vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:37.354301 751 leader_election.cc:304] T e7ca0bd1dbd542de9ee11e6c72522ae0 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:37.356196 1206 tablet_bootstrap.cc:654] T a2c3f3a89b3d47f1b0dfacfac8218805 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:37.360368 898 leader_election.cc:304] T 038ea1ea231e490ba27ecd5430883058 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
I20250114 02:54:37.360677 1207 tablet_bootstrap.cc:492] T e7ca0bd1dbd542de9ee11e6c72522ae0 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
W20250114 02:54:37.361728 898 leader_election.cc:343] T 038ea1ea231e490ba27ecd5430883058 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:37.362138 2447 raft_consensus.cc:2798] T 038ea1ea231e490ba27ecd5430883058 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:37.362473 2447 raft_consensus.cc:491] T 038ea1ea231e490ba27ecd5430883058 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:37.362380 816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "797d6cdf36704c98aa106aac87a945ca" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:37.362777 2447 raft_consensus.cc:3054] T 038ea1ea231e490ba27ecd5430883058 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:37.362906 816 raft_consensus.cc:3054] T 797d6cdf36704c98aa106aac87a945ca P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:37.364411 2504 consensus_queue.cc:1035] T 68981913851243a7a1875fc99efb3c49 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:37.369437 816 raft_consensus.cc:2463] T 797d6cdf36704c98aa106aac87a945ca P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:37.371750 1205 tablet_bootstrap.cc:654] T e7ca0bd1dbd542de9ee11e6c72522ae0 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:37.373348 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "797d6cdf36704c98aa106aac87a945ca" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:37.374994 2372 raft_consensus.cc:2743] T e7ca0bd1dbd542de9ee11e6c72522ae0 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:37.376533 751 leader_election.cc:304] T 97de018391f84b0ebd3130861681b7b5 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
I20250114 02:54:37.386723 2372 raft_consensus.cc:2798] T 97de018391f84b0ebd3130861681b7b5 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:37.387145 2372 raft_consensus.cc:695] T 97de018391f84b0ebd3130861681b7b5 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 LEADER]: Becoming Leader. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Running, Role: LEADER
I20250114 02:54:37.387838 2372 consensus_queue.cc:237] T 97de018391f84b0ebd3130861681b7b5 P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:37.395483 2528 raft_consensus.cc:513] T ed109b49f20642f988161a3df7872419 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:37.421257 2447 raft_consensus.cc:513] T 038ea1ea231e490ba27ecd5430883058 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:37.428181 2406 consensus_queue.cc:1035] T 670dc7ff15d644e18c296eeb2a6b99e1 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:37.444322 2528 leader_election.cc:290] T ed109b49f20642f988161a3df7872419 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Requested vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:37.425872 960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed109b49f20642f988161a3df7872419" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
I20250114 02:54:37.445616 960 raft_consensus.cc:3054] T ed109b49f20642f988161a3df7872419 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:37.452721 2447 leader_election.cc:290] T 038ea1ea231e490ba27ecd5430883058 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Requested vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:37.453470 1099 raft_consensus.cc:2463] T 97de018391f84b0ebd3130861681b7b5 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 1.
I20250114 02:54:37.403278 1207 tablet_bootstrap.cc:654] T e7ca0bd1dbd542de9ee11e6c72522ae0 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
W20250114 02:54:37.409960 1042 leader_election.cc:343] T 3161f75bc4e84000af022faa9804c810 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Not found: Tablet not found: 3161f75bc4e84000af022faa9804c810
I20250114 02:54:37.457042 1042 leader_election.cc:304] T 3161f75bc4e84000af022faa9804c810 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90d6c2c58bb54e0eafb365625ac33871; no voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2
W20250114 02:54:37.457883 1042 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.65:39397, user_credentials={real_user=slave}} blocked reactor thread for 47953.2us
I20250114 02:54:37.458738 2406 raft_consensus.cc:2743] T 3161f75bc4e84000af022faa9804c810 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:37.460696 814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed109b49f20642f988161a3df7872419" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:37.461311 814 raft_consensus.cc:3054] T ed109b49f20642f988161a3df7872419 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:37.494575 1205 tablet_bootstrap.cc:492] T e7ca0bd1dbd542de9ee11e6c72522ae0 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:37.494987 1205 ts_tablet_manager.cc:1397] T e7ca0bd1dbd542de9ee11e6c72522ae0 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.162s user 0.009s sys 0.000s
I20250114 02:54:37.506459 1205 raft_consensus.cc:357] T e7ca0bd1dbd542de9ee11e6c72522ae0 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:37.507385 1205 raft_consensus.cc:383] T e7ca0bd1dbd542de9ee11e6c72522ae0 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:37.507625 1205 raft_consensus.cc:738] T e7ca0bd1dbd542de9ee11e6c72522ae0 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:37.508276 1205 consensus_queue.cc:260] T e7ca0bd1dbd542de9ee11e6c72522ae0 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:37.530081 2513 raft_consensus.cc:491] T e06daadd346d45969ad5efcf03bc0870 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:37.530517 2513 raft_consensus.cc:513] T e06daadd346d45969ad5efcf03bc0870 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:37.532500 2513 leader_election.cc:290] T e06daadd346d45969ad5efcf03bc0870 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:37.536582 2484 raft_consensus.cc:491] T 6d741bfd00784d2d830be3ebe50dc7c4 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:37.537143 2484 raft_consensus.cc:513] T 6d741bfd00784d2d830be3ebe50dc7c4 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:37.540302 2484 leader_election.cc:290] T 6d741bfd00784d2d830be3ebe50dc7c4 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:37.544514 816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06daadd346d45969ad5efcf03bc0870" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:37.545104 816 raft_consensus.cc:2463] T e06daadd346d45969ad5efcf03bc0870 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 0.
I20250114 02:54:37.471572 960 raft_consensus.cc:2463] T ed109b49f20642f988161a3df7872419 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1.
I20250114 02:54:37.549953 1099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d741bfd00784d2d830be3ebe50dc7c4" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
W20250114 02:54:37.551059 753 leader_election.cc:343] T 6d741bfd00784d2d830be3ebe50dc7c4 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:37.551944 1042 leader_election.cc:304] T e06daadd346d45969ad5efcf03bc0870 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:37.552754 2525 raft_consensus.cc:2798] T e06daadd346d45969ad5efcf03bc0870 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:37.553053 2525 raft_consensus.cc:491] T e06daadd346d45969ad5efcf03bc0870 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:37.553328 2525 raft_consensus.cc:3054] T e06daadd346d45969ad5efcf03bc0870 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
W20250114 02:54:37.556886 2558 log.cc:926] Time spent T 97de018391f84b0ebd3130861681b7b5 P 819f9e3412e54d44abb11d8b3ca877c2: Append to log took a long time: real 0.083s user 0.001s sys 0.000s
W20250114 02:54:37.558668 2553 log.cc:926] Time spent T 5d545ac27a7f49faace0655713405cea P 90d6c2c58bb54e0eafb365625ac33871: Append to log took a long time: real 0.083s user 0.001s sys 0.000s
I20250114 02:54:37.567132 814 raft_consensus.cc:2463] T ed109b49f20642f988161a3df7872419 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1.
I20250114 02:54:37.586563 1042 leader_election.cc:304] T ed109b49f20642f988161a3df7872419 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:37.587383 2513 raft_consensus.cc:2798] T ed109b49f20642f988161a3df7872419 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:37.587785 2513 raft_consensus.cc:695] T ed109b49f20642f988161a3df7872419 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Becoming Leader. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Running, Role: LEADER
I20250114 02:54:37.588606 2513 consensus_queue.cc:237] T ed109b49f20642f988161a3df7872419 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:37.605607 960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06daadd346d45969ad5efcf03bc0870" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:37.606202 960 raft_consensus.cc:2463] T e06daadd346d45969ad5efcf03bc0870 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 0.
I20250114 02:54:37.611510 1205 ts_tablet_manager.cc:1428] T e7ca0bd1dbd542de9ee11e6c72522ae0 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.116s user 0.006s sys 0.000s
I20250114 02:54:37.612439 1205 tablet_bootstrap.cc:492] T 45301f8823ed4ec494e792d41f14f71f P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:37.629261 1206 tablet_bootstrap.cc:492] T a2c3f3a89b3d47f1b0dfacfac8218805 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:37.629662 1206 ts_tablet_manager.cc:1397] T a2c3f3a89b3d47f1b0dfacfac8218805 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.362s user 0.012s sys 0.006s
W20250114 02:54:37.635814 898 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.67:45749, user_credentials={real_user=slave}} blocked reactor thread for 273066us
I20250114 02:54:37.639039 1207 tablet_bootstrap.cc:492] T e7ca0bd1dbd542de9ee11e6c72522ae0 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:37.639534 1207 ts_tablet_manager.cc:1397] T e7ca0bd1dbd542de9ee11e6c72522ae0 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.279s user 0.008s sys 0.000s
I20250114 02:54:37.665706 2568 raft_consensus.cc:491] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b303b7df086340a0b5ad6373940ae750)
I20250114 02:54:37.667393 2568 raft_consensus.cc:513] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b303b7df086340a0b5ad6373940ae750" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 45367 } } peers { permanent_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } } peers { permanent_uuid: "87b3db09ac87436e9a60d6de5fedaff5" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } }
I20250114 02:54:37.668949 813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "038ea1ea231e490ba27ecd5430883058" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:37.669507 813 raft_consensus.cc:3054] T 038ea1ea231e490ba27ecd5430883058 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:37.670693 2568 leader_election.cc:290] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b303b7df086340a0b5ad6373940ae750 (127.0.107.126:45367), 687d4ff6d3cb42cf90a965070b0d8a70 (127.0.107.125:46837)
I20250114 02:54:37.671854 556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "87b3db09ac87436e9a60d6de5fedaff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 406 } } ignore_live_leader: false dest_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" is_pre_election: true
I20250114 02:54:37.673972 481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "87b3db09ac87436e9a60d6de5fedaff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 406 } } ignore_live_leader: false dest_uuid: "b303b7df086340a0b5ad6373940ae750" is_pre_election: true
I20250114 02:54:37.675392 2209 leader_election.cc:304] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87b3db09ac87436e9a60d6de5fedaff5; no voters: 687d4ff6d3cb42cf90a965070b0d8a70, b303b7df086340a0b5ad6373940ae750
I20250114 02:54:37.676041 2568 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:54:37.680825 813 raft_consensus.cc:2463] T 038ea1ea231e490ba27ecd5430883058 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:37.631719 1206 raft_consensus.cc:357] T a2c3f3a89b3d47f1b0dfacfac8218805 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:37.692468 1206 raft_consensus.cc:383] T a2c3f3a89b3d47f1b0dfacfac8218805 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:37.692765 1206 raft_consensus.cc:738] T a2c3f3a89b3d47f1b0dfacfac8218805 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:37.693585 1206 consensus_queue.cc:260] T a2c3f3a89b3d47f1b0dfacfac8218805 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:37.721365 2513 raft_consensus.cc:491] T d56255ca806d4e2cac09fbb5e1b1efbd P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:37.721869 2513 raft_consensus.cc:513] T d56255ca806d4e2cac09fbb5e1b1efbd P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:37.723631 2513 leader_election.cc:290] T d56255ca806d4e2cac09fbb5e1b1efbd P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:37.724545 960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d56255ca806d4e2cac09fbb5e1b1efbd" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:37.725813 1100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "038ea1ea231e490ba27ecd5430883058" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:37.731879 1206 ts_tablet_manager.cc:1428] T a2c3f3a89b3d47f1b0dfacfac8218805 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.102s user 0.005s sys 0.001s
I20250114 02:54:37.732755 1206 tablet_bootstrap.cc:492] T 550e84ac4d1b4ef58dfdea6e552ee940 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:37.741860 1207 raft_consensus.cc:357] T e7ca0bd1dbd542de9ee11e6c72522ae0 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:37.742504 1207 raft_consensus.cc:383] T e7ca0bd1dbd542de9ee11e6c72522ae0 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:37.742755 1207 raft_consensus.cc:738] T e7ca0bd1dbd542de9ee11e6c72522ae0 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:37.744253 1207 consensus_queue.cc:260] T e7ca0bd1dbd542de9ee11e6c72522ae0 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:37.746619 1207 ts_tablet_manager.cc:1428] T e7ca0bd1dbd542de9ee11e6c72522ae0 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.107s user 0.002s sys 0.005s
I20250114 02:54:37.752686 2525 raft_consensus.cc:513] T e06daadd346d45969ad5efcf03bc0870 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:37.754478 2525 leader_election.cc:290] T e06daadd346d45969ad5efcf03bc0870 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Requested vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:37.757733 960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06daadd346d45969ad5efcf03bc0870" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
I20250114 02:54:37.757723 1085 tablet_service.cc:1467] Processing CreateTablet for tablet 64d44f95b632489bb362b87baabe80bb (DEFAULT_TABLE table=default.Table_382cf0d973d14925b8424879a1e6d7df [id=c0194382ca85417aa7ed838cef045a95]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:37.758280 960 raft_consensus.cc:3054] T e06daadd346d45969ad5efcf03bc0870 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:37.763286 785 tablet_service.cc:1467] Processing CreateTablet for tablet 64d44f95b632489bb362b87baabe80bb (DEFAULT_TABLE table=default.Table_382cf0d973d14925b8424879a1e6d7df [id=c0194382ca85417aa7ed838cef045a95]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:37.764376 785 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64d44f95b632489bb362b87baabe80bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:37.766054 797 tablet_service.cc:1467] Processing CreateTablet for tablet 409cc36ca0c1437f8c6acda15956371b (DEFAULT_TABLE table=default.Table_c305caaf6328407e9e7e0100432f0e9d [id=0c56c93c919743228d2e35b39fcc805f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:37.766901 797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 409cc36ca0c1437f8c6acda15956371b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:37.769026 1075 tablet_service.cc:1467] Processing CreateTablet for tablet 409cc36ca0c1437f8c6acda15956371b (DEFAULT_TABLE table=default.Table_c305caaf6328407e9e7e0100432f0e9d [id=0c56c93c919743228d2e35b39fcc805f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:37.769078 814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d56255ca806d4e2cac09fbb5e1b1efbd" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:37.769237 786 tablet_service.cc:1467] Processing CreateTablet for tablet 40f2ce33e49b4756ae7fa097a2b4c048 (DEFAULT_TABLE table=default.Table_fb66bdd04429412b9585afd33d5b8876 [id=5ff61b779680460098f62f3aa92ed309]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:37.769824 814 raft_consensus.cc:2388] T d56255ca806d4e2cac09fbb5e1b1efbd P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 90d6c2c58bb54e0eafb365625ac33871 in current term 1: Already voted for candidate 525198593c8e47afaee54e74267b87e3 in this term.
I20250114 02:54:37.770144 786 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40f2ce33e49b4756ae7fa097a2b4c048. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:37.770998 813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06daadd346d45969ad5efcf03bc0870" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:37.771499 813 raft_consensus.cc:3054] T e06daadd346d45969ad5efcf03bc0870 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:37.773032 1084 tablet_service.cc:1467] Processing CreateTablet for tablet 40f2ce33e49b4756ae7fa097a2b4c048 (DEFAULT_TABLE table=default.Table_fb66bdd04429412b9585afd33d5b8876 [id=5ff61b779680460098f62f3aa92ed309]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:37.790035 1086 tablet_service.cc:1467] Processing CreateTablet for tablet 3658b18b42674406ad24bd0b0f79de43 (DEFAULT_TABLE table=default.Table_539d51840e36495ab6185647ab9044e3 [id=35e5cff38891455aa271ec4f1939333e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:37.790397 787 tablet_service.cc:1467] Processing CreateTablet for tablet 3658b18b42674406ad24bd0b0f79de43 (DEFAULT_TABLE table=default.Table_539d51840e36495ab6185647ab9044e3 [id=35e5cff38891455aa271ec4f1939333e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:37.791484 787 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3658b18b42674406ad24bd0b0f79de43. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:37.790763 960 raft_consensus.cc:2463] T e06daadd346d45969ad5efcf03bc0870 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1.
I20250114 02:54:37.793074 813 raft_consensus.cc:2463] T e06daadd346d45969ad5efcf03bc0870 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1.
I20250114 02:54:37.798581 784 tablet_service.cc:1467] Processing CreateTablet for tablet a5676ce60b6b42ea94fc620be5d4ca03 (DEFAULT_TABLE table=default.Table_5e1c3ceefda2466da137a659035f9ae4 [id=ee635d0ed76b46c9b2e139a75296017b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:37.799639 784 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5676ce60b6b42ea94fc620be5d4ca03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:37.806123 1081 tablet_service.cc:1467] Processing CreateTablet for tablet a5676ce60b6b42ea94fc620be5d4ca03 (DEFAULT_TABLE table=default.Table_5e1c3ceefda2466da137a659035f9ae4 [id=ee635d0ed76b46c9b2e139a75296017b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:37.797605 1042 leader_election.cc:304] T d56255ca806d4e2cac09fbb5e1b1efbd P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90d6c2c58bb54e0eafb365625ac33871; no voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2
I20250114 02:54:37.809118 1085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64d44f95b632489bb362b87baabe80bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:37.809484 2513 raft_consensus.cc:3054] T d56255ca806d4e2cac09fbb5e1b1efbd P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:37.812716 1084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40f2ce33e49b4756ae7fa097a2b4c048. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:37.814756 1075 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 409cc36ca0c1437f8c6acda15956371b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:37.818222 1207 tablet_bootstrap.cc:492] T 45301f8823ed4ec494e792d41f14f71f P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:37.819525 2513 raft_consensus.cc:2743] T d56255ca806d4e2cac09fbb5e1b1efbd P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:37.820551 1086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3658b18b42674406ad24bd0b0f79de43. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:37.824960 1042 leader_election.cc:304] T e06daadd346d45969ad5efcf03bc0870 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:37.826278 2510 raft_consensus.cc:2798] T e06daadd346d45969ad5efcf03bc0870 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:37.826727 2510 raft_consensus.cc:695] T e06daadd346d45969ad5efcf03bc0870 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Becoming Leader. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Running, Role: LEADER
I20250114 02:54:37.827471 2510 consensus_queue.cc:237] T e06daadd346d45969ad5efcf03bc0870 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:37.838596 2372 consensus_queue.cc:1035] T 2e17f619ed2745078541cc1c924aff96 P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:37.849066 1206 tablet_bootstrap.cc:654] T 550e84ac4d1b4ef58dfdea6e552ee940 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
W20250114 02:54:37.849830 868 tablet.cc:2367] T a2c3f3a89b3d47f1b0dfacfac8218805 P 819f9e3412e54d44abb11d8b3ca877c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250114 02:54:37.863022 462 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 141cede72b8741a8b0615d7c6bb823bb (table default.Table_4364d1df16d149d2ba10469ce7288d4e [id=13fc996d911b430290e37a94dd44b1ab]). Expected version 1 got 0
I20250114 02:54:37.865147 462 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 30a56fb2d941470590e17f37b90aef26 (table default.Table_0e0710e3f43e4e859ad32662704ee41c [id=66082998119e4c43a666fb43087fc622]). Expected version 1 got 0
I20250114 02:54:37.868825 462 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 38fe8829d98547fb907ef78099df3d7f (table default.Table_60450873174a4c80aa7e2122343d82be [id=9299683349354f879847b7c9d93f6072]). Expected version 1 got 0
I20250114 02:54:37.873785 1206 tablet_bootstrap.cc:492] T 550e84ac4d1b4ef58dfdea6e552ee940 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:37.874166 1206 ts_tablet_manager.cc:1397] T 550e84ac4d1b4ef58dfdea6e552ee940 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.142s user 0.003s sys 0.004s
I20250114 02:54:37.882951 1076 tablet_service.cc:1467] Processing CreateTablet for tablet 30a01ab975704191886c0d7efda96276 (DEFAULT_TABLE table=default.Table_5940a20d29474d1a8dcc1d01e3599699 [id=ad58681da58346d9ad644c7429c37212]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:37.884953 784 tablet_service.cc:1467] Processing CreateTablet for tablet 30a01ab975704191886c0d7efda96276 (DEFAULT_TABLE table=default.Table_5940a20d29474d1a8dcc1d01e3599699 [id=ad58681da58346d9ad644c7429c37212]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:37.885931 784 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30a01ab975704191886c0d7efda96276. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:37.886706 1081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5676ce60b6b42ea94fc620be5d4ca03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:37.889138 1079 tablet_service.cc:1467] Processing CreateTablet for tablet 7213d1cc53034192ba2a1822ac1fa929 (DEFAULT_TABLE table=default.Table_c50c01c8bc7e44a3bd139bc573c70122 [id=9171c778af084ef2bb6032e09a096769]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:37.891247 785 tablet_service.cc:1467] Processing CreateTablet for tablet 7213d1cc53034192ba2a1822ac1fa929 (DEFAULT_TABLE table=default.Table_c50c01c8bc7e44a3bd139bc573c70122 [id=9171c778af084ef2bb6032e09a096769]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:37.892346 785 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7213d1cc53034192ba2a1822ac1fa929. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:37.898239 1079 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7213d1cc53034192ba2a1822ac1fa929. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:37.901214 1076 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30a01ab975704191886c0d7efda96276. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:37.901346 1206 raft_consensus.cc:357] T 550e84ac4d1b4ef58dfdea6e552ee940 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:37.901994 1206 raft_consensus.cc:383] T 550e84ac4d1b4ef58dfdea6e552ee940 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:37.902216 1206 raft_consensus.cc:738] T 550e84ac4d1b4ef58dfdea6e552ee940 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:37.902746 1206 consensus_queue.cc:260] T 550e84ac4d1b4ef58dfdea6e552ee940 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
W20250114 02:54:37.916335 1013 tablet.cc:2367] T e7ca0bd1dbd542de9ee11e6c72522ae0 P 525198593c8e47afaee54e74267b87e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250114 02:54:37.921732 934 tablet_service.cc:1467] Processing CreateTablet for tablet 64d44f95b632489bb362b87baabe80bb (DEFAULT_TABLE table=default.Table_382cf0d973d14925b8424879a1e6d7df [id=c0194382ca85417aa7ed838cef045a95]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:37.922346 932 tablet_service.cc:1467] Processing CreateTablet for tablet 409cc36ca0c1437f8c6acda15956371b (DEFAULT_TABLE table=default.Table_c305caaf6328407e9e7e0100432f0e9d [id=0c56c93c919743228d2e35b39fcc805f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:37.922727 934 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64d44f95b632489bb362b87baabe80bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:37.923197 932 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 409cc36ca0c1437f8c6acda15956371b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:37.924069 2500 raft_consensus.cc:491] T e7ca0bd1dbd542de9ee11e6c72522ae0 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:37.924907 939 tablet_service.cc:1467] Processing CreateTablet for tablet 40f2ce33e49b4756ae7fa097a2b4c048 (DEFAULT_TABLE table=default.Table_fb66bdd04429412b9585afd33d5b8876 [id=5ff61b779680460098f62f3aa92ed309]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:37.925720 939 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40f2ce33e49b4756ae7fa097a2b4c048. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:37.928857 1206 ts_tablet_manager.cc:1428] T 550e84ac4d1b4ef58dfdea6e552ee940 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.054s user 0.007s sys 0.000s
I20250114 02:54:37.929730 1206 tablet_bootstrap.cc:492] T 46eb13912f754bf1b1fe9d0ad429c3c1 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:37.931638 936 tablet_service.cc:1467] Processing CreateTablet for tablet 3658b18b42674406ad24bd0b0f79de43 (DEFAULT_TABLE table=default.Table_539d51840e36495ab6185647ab9044e3 [id=35e5cff38891455aa271ec4f1939333e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:37.937055 1207 tablet_bootstrap.cc:654] T 45301f8823ed4ec494e792d41f14f71f P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:37.938786 935 tablet_service.cc:1467] Processing CreateTablet for tablet 30a01ab975704191886c0d7efda96276 (DEFAULT_TABLE table=default.Table_5940a20d29474d1a8dcc1d01e3599699 [id=ad58681da58346d9ad644c7429c37212]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:37.939703 935 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30a01ab975704191886c0d7efda96276. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:37.940907 933 tablet_service.cc:1467] Processing CreateTablet for tablet a5676ce60b6b42ea94fc620be5d4ca03 (DEFAULT_TABLE table=default.Table_5e1c3ceefda2466da137a659035f9ae4 [id=ee635d0ed76b46c9b2e139a75296017b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:37.941794 933 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5676ce60b6b42ea94fc620be5d4ca03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:37.924481 2500 raft_consensus.cc:513] T e7ca0bd1dbd542de9ee11e6c72522ae0 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:37.943686 2510 consensus_queue.cc:1035] T 0b8361cd538046e0bcb4aac821748f67 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:37.938787 931 tablet_service.cc:1467] Processing CreateTablet for tablet 7213d1cc53034192ba2a1822ac1fa929 (DEFAULT_TABLE table=default.Table_c50c01c8bc7e44a3bd139bc573c70122 [id=9171c778af084ef2bb6032e09a096769]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:54:37.945459 931 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7213d1cc53034192ba2a1822ac1fa929. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:37.949309 936 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3658b18b42674406ad24bd0b0f79de43. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:37.953064 1206 tablet_bootstrap.cc:654] T 46eb13912f754bf1b1fe9d0ad429c3c1 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:37.958946 2500 leader_election.cc:290] T e7ca0bd1dbd542de9ee11e6c72522ae0 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:37.971323 1206 tablet_bootstrap.cc:492] T 46eb13912f754bf1b1fe9d0ad429c3c1 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:37.974133 1206 ts_tablet_manager.cc:1397] T 46eb13912f754bf1b1fe9d0ad429c3c1 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.045s user 0.009s sys 0.000s
I20250114 02:54:37.977448 1206 raft_consensus.cc:357] T 46eb13912f754bf1b1fe9d0ad429c3c1 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:37.978067 1206 raft_consensus.cc:383] T 46eb13912f754bf1b1fe9d0ad429c3c1 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:37.978307 1206 raft_consensus.cc:738] T 46eb13912f754bf1b1fe9d0ad429c3c1 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:37.978852 1206 consensus_queue.cc:260] T 46eb13912f754bf1b1fe9d0ad429c3c1 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:37.980767 960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d741bfd00784d2d830be3ebe50dc7c4" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:37.981173 2510 consensus_queue.cc:1035] T 0b8361cd538046e0bcb4aac821748f67 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:38.001124 462 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet d2a7b2bee5c1440fb705b172c18a9247 (table default.Table_3c0c8ba6eec34be4a6ea569d66038106 [id=4d2f7fabf8a44155a586a2b79a8fca53]). Expected version 1 got 0
I20250114 02:54:37.986317 898 leader_election.cc:304] T 797d6cdf36704c98aa106aac87a945ca P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
I20250114 02:54:38.003319 898 leader_election.cc:304] T 038ea1ea231e490ba27ecd5430883058 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
I20250114 02:54:38.003940 2556 raft_consensus.cc:2798] T 797d6cdf36704c98aa106aac87a945ca P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:38.006274 2500 raft_consensus.cc:2798] T 038ea1ea231e490ba27ecd5430883058 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:38.006704 2500 raft_consensus.cc:695] T 038ea1ea231e490ba27ecd5430883058 P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Becoming Leader. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Running, Role: LEADER
I20250114 02:54:38.006788 2556 raft_consensus.cc:695] T 797d6cdf36704c98aa106aac87a945ca P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Becoming Leader. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Running, Role: LEADER
I20250114 02:54:38.007462 2556 consensus_queue.cc:237] T 797d6cdf36704c98aa106aac87a945ca P 525198593c8e47afaee54e74267b87e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
W20250114 02:54:38.008656 898 leader_election.cc:343] T 797d6cdf36704c98aa106aac87a945ca P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:38.006814 2504 consensus_queue.cc:1035] T a2d68eccaaa64511a9ae36f2b07a7b59 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20250114 02:54:38.007462 2500 consensus_queue.cc:237] T 038ea1ea231e490ba27ecd5430883058 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:38.015271 462 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet a52d6749ff834b62a8e20a2f45935240 (table default.Table_1c9b7896941e4289b1c12e20c210c2c1 [id=b3ee15b9f10f4d489edbde3af9034d25]). Expected version 1 got 0
W20250114 02:54:38.017879 898 leader_election.cc:343] T 038ea1ea231e490ba27ecd5430883058 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:38.021621 2449 consensus_queue.cc:1035] T 68981913851243a7a1875fc99efb3c49 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:38.023352 2449 consensus_queue.cc:1035] T a2d68eccaaa64511a9ae36f2b07a7b59 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:38.025324 1206 ts_tablet_manager.cc:1428] T 46eb13912f754bf1b1fe9d0ad429c3c1 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.051s user 0.006s sys 0.000s
I20250114 02:54:38.040207 813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7ca0bd1dbd542de9ee11e6c72522ae0" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:38.041249 813 raft_consensus.cc:2463] T e7ca0bd1dbd542de9ee11e6c72522ae0 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:38.043635 1099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7ca0bd1dbd542de9ee11e6c72522ae0" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:38.044147 1099 raft_consensus.cc:2463] T e7ca0bd1dbd542de9ee11e6c72522ae0 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
W20250114 02:54:38.046730 751 leader_election.cc:343] T 6d741bfd00784d2d830be3ebe50dc7c4 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:38.047072 751 leader_election.cc:304] T 6d741bfd00784d2d830be3ebe50dc7c4 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:38.048146 2532 raft_consensus.cc:2743] T 6d741bfd00784d2d830be3ebe50dc7c4 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:38.050894 1206 tablet_bootstrap.cc:492] T 2d4eaf90be6d497d8a3c4d9ac418f4e9 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:38.059406 462 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 68981913851243a7a1875fc99efb3c49 (table default.Table_5e2861a7715a47039e64f1a717c57997 [id=683a61c5af09453e8027e9cb5b5b80a7]). Expected version 1 got 0
I20250114 02:54:38.060173 1207 tablet_bootstrap.cc:492] T 45301f8823ed4ec494e792d41f14f71f P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:38.060003 462 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 719188f639724851a5fc2b37cb60fdbb (table default.Table_8f8a18d7868c44bbb4d66a0692b08186 [id=e6292728826f437cb75c524a8aef2e87]). Expected version 1 got 0
I20250114 02:54:38.060626 1207 ts_tablet_manager.cc:1397] T 45301f8823ed4ec494e792d41f14f71f P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.243s user 0.010s sys 0.000s
I20250114 02:54:38.061261 462 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet df045d80d8ce4ff194a6d2039a44bce6 (table default.Table_2befe573a994473abe0556ba7e713686 [id=da0258578bb34ae29551fc593bf7d0b5]). Expected version 1 got 0
I20250114 02:54:38.062397 462 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 5d545ac27a7f49faace0655713405cea (table default.Table_b4dc2961391c43fcb08fa062d4e45b3c [id=491b34b03ce14bd9be16e04915468928]). Expected version 1 got 0
I20250114 02:54:38.062816 2532 consensus_queue.cc:1035] T 2e17f619ed2745078541cc1c924aff96 P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:54:38.084375 2532 raft_consensus.cc:491] T 550e84ac4d1b4ef58dfdea6e552ee940 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:38.084797 2532 raft_consensus.cc:513] T 550e84ac4d1b4ef58dfdea6e552ee940 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:38.087715 2532 leader_election.cc:290] T 550e84ac4d1b4ef58dfdea6e552ee940 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:38.089293 1100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550e84ac4d1b4ef58dfdea6e552ee940" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
W20250114 02:54:38.090688 753 leader_election.cc:343] T 550e84ac4d1b4ef58dfdea6e552ee940 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:38.129930 2513 consensus_queue.cc:1035] T be7aad41ccdd4ef9bcfe8a985155708f P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:38.132390 463 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet edc69b66bea54625a4b15867b75986b7 (table default.Table_0a0bda55a4a9476c98d961822b8a19bd [id=fb2be5c1ca9444a197f3315ea9c59b09]). Expected version 1 got 0
I20250114 02:54:38.146852 463 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 141cede72b8741a8b0615d7c6bb823bb (table default.Table_4364d1df16d149d2ba10469ce7288d4e [id=13fc996d911b430290e37a94dd44b1ab]). Expected version 1 got 0
I20250114 02:54:38.147492 1206 tablet_bootstrap.cc:654] T 2d4eaf90be6d497d8a3c4d9ac418f4e9 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:38.147989 463 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 30a56fb2d941470590e17f37b90aef26 (table default.Table_0e0710e3f43e4e859ad32662704ee41c [id=66082998119e4c43a666fb43087fc622]). Expected version 1 got 0
I20250114 02:54:38.150995 463 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 38fe8829d98547fb907ef78099df3d7f (table default.Table_60450873174a4c80aa7e2122343d82be [id=9299683349354f879847b7c9d93f6072]). Expected version 1 got 0
I20250114 02:54:38.151234 1207 raft_consensus.cc:357] T 45301f8823ed4ec494e792d41f14f71f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:38.151871 1207 raft_consensus.cc:383] T 45301f8823ed4ec494e792d41f14f71f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:38.163607 463 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet d2a7b2bee5c1440fb705b172c18a9247 (table default.Table_3c0c8ba6eec34be4a6ea569d66038106 [id=4d2f7fabf8a44155a586a2b79a8fca53]). Expected version 1 got 0
I20250114 02:54:38.184868 2513 consensus_queue.cc:1035] T be7aad41ccdd4ef9bcfe8a985155708f P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:38.152086 1207 raft_consensus.cc:738] T 45301f8823ed4ec494e792d41f14f71f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:38.199277 1207 consensus_queue.cc:260] T 45301f8823ed4ec494e792d41f14f71f P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:38.203893 1207 ts_tablet_manager.cc:1428] T 45301f8823ed4ec494e792d41f14f71f P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.143s user 0.005s sys 0.000s
I20250114 02:54:38.220824 463 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet a52d6749ff834b62a8e20a2f45935240 (table default.Table_1c9b7896941e4289b1c12e20c210c2c1 [id=b3ee15b9f10f4d489edbde3af9034d25]). Expected version 1 got 0
I20250114 02:54:38.224931 463 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet df045d80d8ce4ff194a6d2039a44bce6 (table default.Table_2befe573a994473abe0556ba7e713686 [id=da0258578bb34ae29551fc593bf7d0b5]). Expected version 1 got 0
I20250114 02:54:38.218967 1207 tablet_bootstrap.cc:492] T 797d6cdf36704c98aa106aac87a945ca P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:38.227066 2447 consensus_queue.cc:1035] T 612987f3e4ea4a02932ceefa0d740425 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:38.231272 958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550e84ac4d1b4ef58dfdea6e552ee940" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:38.231689 1207 tablet_bootstrap.cc:654] T 797d6cdf36704c98aa106aac87a945ca P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:38.239886 898 leader_election.cc:304] T e7ca0bd1dbd542de9ee11e6c72522ae0 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
I20250114 02:54:38.241444 463 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 719188f639724851a5fc2b37cb60fdbb (table default.Table_8f8a18d7868c44bbb4d66a0692b08186 [id=e6292728826f437cb75c524a8aef2e87]). Expected version 1 got 0
I20250114 02:54:38.241922 2447 raft_consensus.cc:2798] T e7ca0bd1dbd542de9ee11e6c72522ae0 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:38.242287 2447 raft_consensus.cc:491] T e7ca0bd1dbd542de9ee11e6c72522ae0 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:38.242647 2447 raft_consensus.cc:3054] T e7ca0bd1dbd542de9ee11e6c72522ae0 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:38.243252 463 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 5d545ac27a7f49faace0655713405cea (table default.Table_b4dc2961391c43fcb08fa062d4e45b3c [id=491b34b03ce14bd9be16e04915468928]). Expected version 1 got 0
I20250114 02:54:38.270473 1206 tablet_bootstrap.cc:492] T 2d4eaf90be6d497d8a3c4d9ac418f4e9 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:38.270961 1206 ts_tablet_manager.cc:1397] T 2d4eaf90be6d497d8a3c4d9ac418f4e9 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.221s user 0.012s sys 0.002s
I20250114 02:54:38.283394 2573 consensus_queue.cc:1035] T 612987f3e4ea4a02932ceefa0d740425 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:38.287169 1206 raft_consensus.cc:357] T 2d4eaf90be6d497d8a3c4d9ac418f4e9 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:38.288969 2598 raft_consensus.cc:491] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b303b7df086340a0b5ad6373940ae750)
I20250114 02:54:38.289464 2598 raft_consensus.cc:513] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b303b7df086340a0b5ad6373940ae750" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 45367 } } peers { permanent_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } } peers { permanent_uuid: "87b3db09ac87436e9a60d6de5fedaff5" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } }
I20250114 02:54:38.290535 2447 raft_consensus.cc:513] T e7ca0bd1dbd542de9ee11e6c72522ae0 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:38.292300 2447 leader_election.cc:290] T e7ca0bd1dbd542de9ee11e6c72522ae0 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Requested vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:38.292608 481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "87b3db09ac87436e9a60d6de5fedaff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 406 } } ignore_live_leader: false dest_uuid: "b303b7df086340a0b5ad6373940ae750" is_pre_election: true
I20250114 02:54:38.306036 1206 raft_consensus.cc:383] T 2d4eaf90be6d497d8a3c4d9ac418f4e9 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:38.308522 1206 raft_consensus.cc:738] T 2d4eaf90be6d497d8a3c4d9ac418f4e9 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:38.309943 1206 consensus_queue.cc:260] T 2d4eaf90be6d497d8a3c4d9ac418f4e9 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:38.328791 2598 leader_election.cc:290] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b303b7df086340a0b5ad6373940ae750 (127.0.107.126:45367), 687d4ff6d3cb42cf90a965070b0d8a70 (127.0.107.125:46837)
I20250114 02:54:38.329720 556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "87b3db09ac87436e9a60d6de5fedaff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 406 } } ignore_live_leader: false dest_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" is_pre_election: true
I20250114 02:54:38.330983 2204 leader_election.cc:304] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87b3db09ac87436e9a60d6de5fedaff5; no voters: 687d4ff6d3cb42cf90a965070b0d8a70, b303b7df086340a0b5ad6373940ae750
I20250114 02:54:38.332348 2598 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:54:38.340009 464 catalog_manager.cc:5526] T 141cede72b8741a8b0615d7c6bb823bb P 525198593c8e47afaee54e74267b87e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67). New cstate: current_term: 1 leader_uuid: "90d6c2c58bb54e0eafb365625ac33871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } }
I20250114 02:54:38.341110 464 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 141cede72b8741a8b0615d7c6bb823bb (table default.Table_4364d1df16d149d2ba10469ce7288d4e [id=13fc996d911b430290e37a94dd44b1ab]). Expected version 1 got 0
I20250114 02:54:38.342090 464 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 38fe8829d98547fb907ef78099df3d7f (table default.Table_60450873174a4c80aa7e2122343d82be [id=9299683349354f879847b7c9d93f6072]). Expected version 1 got 0
I20250114 02:54:38.342631 1207 tablet_bootstrap.cc:492] T 797d6cdf36704c98aa106aac87a945ca P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:38.343021 1207 ts_tablet_manager.cc:1397] T 797d6cdf36704c98aa106aac87a945ca P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.124s user 0.009s sys 0.000s
I20250114 02:54:38.342929 2449 consensus_queue.cc:1035] T d56255ca806d4e2cac09fbb5e1b1efbd P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:38.343755 464 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet d2a7b2bee5c1440fb705b172c18a9247 (table default.Table_3c0c8ba6eec34be4a6ea569d66038106 [id=4d2f7fabf8a44155a586a2b79a8fca53]). Expected version 1 got 0
I20250114 02:54:38.345420 1207 raft_consensus.cc:357] T 797d6cdf36704c98aa106aac87a945ca P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:38.346326 1207 raft_consensus.cc:383] T 797d6cdf36704c98aa106aac87a945ca P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:38.346635 1207 raft_consensus.cc:738] T 797d6cdf36704c98aa106aac87a945ca P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:38.347263 1207 consensus_queue.cc:260] T 797d6cdf36704c98aa106aac87a945ca P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:38.361636 1205 tablet_bootstrap.cc:654] T 45301f8823ed4ec494e792d41f14f71f P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:38.362129 464 catalog_manager.cc:5526] T f40c66314342453b898da3ac0e7e86bf P 525198593c8e47afaee54e74267b87e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 525198593c8e47afaee54e74267b87e3 (127.0.107.66). New cstate: current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:38.369336 2510 consensus_queue.cc:1035] T ed109b49f20642f988161a3df7872419 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:54:38.370990 464 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet a52d6749ff834b62a8e20a2f45935240 (table default.Table_1c9b7896941e4289b1c12e20c210c2c1 [id=b3ee15b9f10f4d489edbde3af9034d25]). Expected version 1 got 0
I20250114 02:54:38.379290 464 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 719188f639724851a5fc2b37cb60fdbb (table default.Table_8f8a18d7868c44bbb4d66a0692b08186 [id=e6292728826f437cb75c524a8aef2e87]). Expected version 1 got 0
I20250114 02:54:38.385712 1206 ts_tablet_manager.cc:1428] T 2d4eaf90be6d497d8a3c4d9ac418f4e9 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.114s user 0.004s sys 0.000s
I20250114 02:54:38.386511 1206 tablet_bootstrap.cc:492] T ab9a85db127b4d82ae1161717ed6006a P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:38.388924 2532 consensus_queue.cc:1035] T 97de018391f84b0ebd3130861681b7b5 P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:54:38.387732 464 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet df045d80d8ce4ff194a6d2039a44bce6 (table default.Table_2befe573a994473abe0556ba7e713686 [id=da0258578bb34ae29551fc593bf7d0b5]). Expected version 1 got 0
I20250114 02:54:38.392450 464 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 30a56fb2d941470590e17f37b90aef26 (table default.Table_0e0710e3f43e4e859ad32662704ee41c [id=66082998119e4c43a666fb43087fc622]). Expected version 1 got 0
I20250114 02:54:38.402227 1207 ts_tablet_manager.cc:1428] T 797d6cdf36704c98aa106aac87a945ca P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.059s user 0.005s sys 0.000s
I20250114 02:54:38.403539 1207 tablet_bootstrap.cc:492] T 038ea1ea231e490ba27ecd5430883058 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
W20250114 02:54:38.414080 456 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 141cede72b8741a8b0615d7c6bb823bb (table default.Table_4364d1df16d149d2ba10469ce7288d4e [id=13fc996d911b430290e37a94dd44b1ab], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 141cede72b8741a8b0615d7c6bb823bb consensus state has no leader
I20250114 02:54:38.440436 2525 consensus_queue.cc:1035] T ed109b49f20642f988161a3df7872419 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250114 02:54:38.444777 751 leader_election.cc:343] T 550e84ac4d1b4ef58dfdea6e552ee940 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:38.445161 751 leader_election.cc:304] T 550e84ac4d1b4ef58dfdea6e552ee940 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
W20250114 02:54:38.448676 1158 tablet.cc:2367] T 45301f8823ed4ec494e792d41f14f71f P 90d6c2c58bb54e0eafb365625ac33871: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250114 02:54:38.448817 2532 raft_consensus.cc:2743] T 550e84ac4d1b4ef58dfdea6e552ee940 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:38.454579 811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7ca0bd1dbd542de9ee11e6c72522ae0" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:38.455142 811 raft_consensus.cc:3054] T e7ca0bd1dbd542de9ee11e6c72522ae0 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:38.456533 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7ca0bd1dbd542de9ee11e6c72522ae0" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:38.457775 1106 raft_consensus.cc:3054] T e7ca0bd1dbd542de9ee11e6c72522ae0 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:38.463799 1106 raft_consensus.cc:2463] T e7ca0bd1dbd542de9ee11e6c72522ae0 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:38.471000 811 raft_consensus.cc:2463] T e7ca0bd1dbd542de9ee11e6c72522ae0 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:38.476554 469 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet df045d80d8ce4ff194a6d2039a44bce6 (table default.Table_2befe573a994473abe0556ba7e713686 [id=da0258578bb34ae29551fc593bf7d0b5]). Expected version 1 got 0
I20250114 02:54:38.465142 1207 tablet_bootstrap.cc:654] T 038ea1ea231e490ba27ecd5430883058 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:38.493356 1205 tablet_bootstrap.cc:492] T 45301f8823ed4ec494e792d41f14f71f P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:38.493758 1205 ts_tablet_manager.cc:1397] T 45301f8823ed4ec494e792d41f14f71f P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.881s user 0.015s sys 0.003s
I20250114 02:54:38.477119 469 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet 30a56fb2d941470590e17f37b90aef26 (table default.Table_0e0710e3f43e4e859ad32662704ee41c [id=66082998119e4c43a666fb43087fc622]). Expected version 1 got 0
I20250114 02:54:38.496920 469 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet 141cede72b8741a8b0615d7c6bb823bb (table default.Table_4364d1df16d149d2ba10469ce7288d4e [id=13fc996d911b430290e37a94dd44b1ab]). Expected version 1 got 0
I20250114 02:54:38.499023 469 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet a52d6749ff834b62a8e20a2f45935240 (table default.Table_1c9b7896941e4289b1c12e20c210c2c1 [id=b3ee15b9f10f4d489edbde3af9034d25]). Expected version 1 got 0
I20250114 02:54:38.495860 1205 raft_consensus.cc:357] T 45301f8823ed4ec494e792d41f14f71f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:38.510540 1205 raft_consensus.cc:383] T 45301f8823ed4ec494e792d41f14f71f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:38.514025 1205 raft_consensus.cc:738] T 45301f8823ed4ec494e792d41f14f71f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:38.514573 1205 consensus_queue.cc:260] T 45301f8823ed4ec494e792d41f14f71f P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:38.515092 469 catalog_manager.cc:5526] T d2a7b2bee5c1440fb705b172c18a9247 P 90d6c2c58bb54e0eafb365625ac33871 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67). New cstate: current_term: 1 leader_uuid: "90d6c2c58bb54e0eafb365625ac33871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:38.516304 469 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet d2a7b2bee5c1440fb705b172c18a9247 (table default.Table_3c0c8ba6eec34be4a6ea569d66038106 [id=4d2f7fabf8a44155a586a2b79a8fca53]). Expected version 1 got 0
I20250114 02:54:38.517254 469 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet 38fe8829d98547fb907ef78099df3d7f (table default.Table_60450873174a4c80aa7e2122343d82be [id=9299683349354f879847b7c9d93f6072]). Expected version 1 got 0
I20250114 02:54:38.518473 469 catalog_manager.cc:5618] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749) does not have the latest schema for tablet 719188f639724851a5fc2b37cb60fdbb (table default.Table_8f8a18d7868c44bbb4d66a0692b08186 [id=e6292728826f437cb75c524a8aef2e87]). Expected version 1 got 0
I20250114 02:54:38.524663 2447 consensus_queue.cc:1035] T d56255ca806d4e2cac09fbb5e1b1efbd P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:38.531853 1206 tablet_bootstrap.cc:654] T ab9a85db127b4d82ae1161717ed6006a P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:38.532577 1207 tablet_bootstrap.cc:492] T 038ea1ea231e490ba27ecd5430883058 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:38.533008 1207 ts_tablet_manager.cc:1397] T 038ea1ea231e490ba27ecd5430883058 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.130s user 0.010s sys 0.000s
I20250114 02:54:38.536082 1207 raft_consensus.cc:357] T 038ea1ea231e490ba27ecd5430883058 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:38.544872 1207 raft_consensus.cc:383] T 038ea1ea231e490ba27ecd5430883058 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:38.545177 1207 raft_consensus.cc:738] T 038ea1ea231e490ba27ecd5430883058 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:38.545758 1207 consensus_queue.cc:260] T 038ea1ea231e490ba27ecd5430883058 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:38.553342 1205 ts_tablet_manager.cc:1428] T 45301f8823ed4ec494e792d41f14f71f P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.059s user 0.007s sys 0.000s
I20250114 02:54:38.554672 2532 raft_consensus.cc:491] T 2d4eaf90be6d497d8a3c4d9ac418f4e9 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:38.555123 2532 raft_consensus.cc:513] T 2d4eaf90be6d497d8a3c4d9ac418f4e9 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:38.556927 2532 leader_election.cc:290] T 2d4eaf90be6d497d8a3c4d9ac418f4e9 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:38.559223 1103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d4eaf90be6d497d8a3c4d9ac418f4e9" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
W20250114 02:54:38.560467 753 leader_election.cc:343] T 2d4eaf90be6d497d8a3c4d9ac418f4e9 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
W20250114 02:54:38.568329 898 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.65:39397, user_credentials={real_user=slave}} blocked reactor thread for 43849.6us
I20250114 02:54:38.572714 898 leader_election.cc:304] T e7ca0bd1dbd542de9ee11e6c72522ae0 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
I20250114 02:54:38.573355 2447 raft_consensus.cc:2798] T e7ca0bd1dbd542de9ee11e6c72522ae0 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:38.573992 2447 raft_consensus.cc:695] T e7ca0bd1dbd542de9ee11e6c72522ae0 P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Becoming Leader. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Running, Role: LEADER
I20250114 02:54:38.574736 2447 consensus_queue.cc:237] T e7ca0bd1dbd542de9ee11e6c72522ae0 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:38.579149 1206 tablet_bootstrap.cc:492] T ab9a85db127b4d82ae1161717ed6006a P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:38.579633 1206 ts_tablet_manager.cc:1397] T ab9a85db127b4d82ae1161717ed6006a P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.193s user 0.007s sys 0.003s
I20250114 02:54:38.582238 1206 raft_consensus.cc:357] T ab9a85db127b4d82ae1161717ed6006a P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:38.582835 1205 tablet_bootstrap.cc:492] T 2d4eaf90be6d497d8a3c4d9ac418f4e9 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:38.589047 2510 consensus_queue.cc:1035] T e06daadd346d45969ad5efcf03bc0870 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:38.598207 1207 ts_tablet_manager.cc:1428] T 038ea1ea231e490ba27ecd5430883058 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.065s user 0.007s sys 0.000s
I20250114 02:54:38.599167 1207 tablet_bootstrap.cc:492] T ab9a85db127b4d82ae1161717ed6006a P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:38.601212 1205 tablet_bootstrap.cc:654] T 2d4eaf90be6d497d8a3c4d9ac418f4e9 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:38.603969 1206 raft_consensus.cc:383] T ab9a85db127b4d82ae1161717ed6006a P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:38.606274 2532 raft_consensus.cc:491] T 46eb13912f754bf1b1fe9d0ad429c3c1 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:38.609872 1206 raft_consensus.cc:738] T ab9a85db127b4d82ae1161717ed6006a P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:38.609928 2532 raft_consensus.cc:513] T 46eb13912f754bf1b1fe9d0ad429c3c1 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:38.612103 2532 leader_election.cc:290] T 46eb13912f754bf1b1fe9d0ad429c3c1 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:38.613750 1100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46eb13912f754bf1b1fe9d0ad429c3c1" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
W20250114 02:54:38.614876 753 leader_election.cc:343] T 46eb13912f754bf1b1fe9d0ad429c3c1 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:38.613637 1206 consensus_queue.cc:260] T ab9a85db127b4d82ae1161717ed6006a P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:38.621591 1206 ts_tablet_manager.cc:1428] T ab9a85db127b4d82ae1161717ed6006a P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.042s user 0.005s sys 0.000s
I20250114 02:54:38.628890 1206 tablet_bootstrap.cc:492] T ddad7584337345109ce102702008ab37 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:38.632186 2525 consensus_queue.cc:1035] T e06daadd346d45969ad5efcf03bc0870 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:38.638257 1206 tablet_bootstrap.cc:654] T ddad7584337345109ce102702008ab37 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:38.640893 468 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 141cede72b8741a8b0615d7c6bb823bb (table default.Table_4364d1df16d149d2ba10469ce7288d4e [id=13fc996d911b430290e37a94dd44b1ab]). Expected version 1 got 0
I20250114 02:54:38.641462 468 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 30a56fb2d941470590e17f37b90aef26 (table default.Table_0e0710e3f43e4e859ad32662704ee41c [id=66082998119e4c43a666fb43087fc622]). Expected version 1 got 0
I20250114 02:54:38.662043 1207 tablet_bootstrap.cc:654] T ab9a85db127b4d82ae1161717ed6006a P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:38.663625 1205 tablet_bootstrap.cc:492] T 2d4eaf90be6d497d8a3c4d9ac418f4e9 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:38.665241 1205 ts_tablet_manager.cc:1397] T 2d4eaf90be6d497d8a3c4d9ac418f4e9 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.083s user 0.011s sys 0.000s
I20250114 02:54:38.667013 2510 raft_consensus.cc:491] T 45301f8823ed4ec494e792d41f14f71f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:38.667467 2510 raft_consensus.cc:513] T 45301f8823ed4ec494e792d41f14f71f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:38.667614 1205 raft_consensus.cc:357] T 2d4eaf90be6d497d8a3c4d9ac418f4e9 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:38.668838 468 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet a52d6749ff834b62a8e20a2f45935240 (table default.Table_1c9b7896941e4289b1c12e20c210c2c1 [id=b3ee15b9f10f4d489edbde3af9034d25]). Expected version 1 got 0
I20250114 02:54:38.669327 2510 leader_election.cc:290] T 45301f8823ed4ec494e792d41f14f71f P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:38.669749 468 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 68981913851243a7a1875fc99efb3c49 (table default.Table_5e2861a7715a47039e64f1a717c57997 [id=683a61c5af09453e8027e9cb5b5b80a7]). Expected version 1 got 0
I20250114 02:54:38.670246 468 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 719188f639724851a5fc2b37cb60fdbb (table default.Table_8f8a18d7868c44bbb4d66a0692b08186 [id=e6292728826f437cb75c524a8aef2e87]). Expected version 1 got 0
I20250114 02:54:38.670815 816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45301f8823ed4ec494e792d41f14f71f" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:38.671399 816 raft_consensus.cc:2463] T 45301f8823ed4ec494e792d41f14f71f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 0.
I20250114 02:54:38.671749 1206 tablet_bootstrap.cc:492] T ddad7584337345109ce102702008ab37 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:38.672473 1042 leader_election.cc:304] T 45301f8823ed4ec494e792d41f14f71f P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:38.673280 2510 raft_consensus.cc:2798] T 45301f8823ed4ec494e792d41f14f71f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:38.673602 2510 raft_consensus.cc:491] T 45301f8823ed4ec494e792d41f14f71f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:38.673899 2510 raft_consensus.cc:3054] T 45301f8823ed4ec494e792d41f14f71f P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:38.672281 959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45301f8823ed4ec494e792d41f14f71f" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:38.681030 2510 raft_consensus.cc:513] T 45301f8823ed4ec494e792d41f14f71f P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:38.681842 1205 raft_consensus.cc:383] T 2d4eaf90be6d497d8a3c4d9ac418f4e9 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:38.682971 2532 raft_consensus.cc:491] T a2c3f3a89b3d47f1b0dfacfac8218805 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:38.683342 1206 ts_tablet_manager.cc:1397] T ddad7584337345109ce102702008ab37 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.055s user 0.004s sys 0.006s
I20250114 02:54:38.683449 2532 raft_consensus.cc:513] T a2c3f3a89b3d47f1b0dfacfac8218805 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:38.685376 2532 leader_election.cc:290] T a2c3f3a89b3d47f1b0dfacfac8218805 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:38.685591 1206 raft_consensus.cc:357] T ddad7584337345109ce102702008ab37 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:38.686254 1206 raft_consensus.cc:383] T ddad7584337345109ce102702008ab37 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:38.686513 1206 raft_consensus.cc:738] T ddad7584337345109ce102702008ab37 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:38.687371 1100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2c3f3a89b3d47f1b0dfacfac8218805" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
W20250114 02:54:38.688859 753 leader_election.cc:343] T a2c3f3a89b3d47f1b0dfacfac8218805 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:38.689616 816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45301f8823ed4ec494e792d41f14f71f" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:38.690382 816 raft_consensus.cc:3054] T 45301f8823ed4ec494e792d41f14f71f P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:38.692358 2510 leader_election.cc:290] T 45301f8823ed4ec494e792d41f14f71f P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Requested vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:38.692796 468 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet df045d80d8ce4ff194a6d2039a44bce6 (table default.Table_2befe573a994473abe0556ba7e713686 [id=da0258578bb34ae29551fc593bf7d0b5]). Expected version 1 got 0
I20250114 02:54:38.693397 468 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet d2a7b2bee5c1440fb705b172c18a9247 (table default.Table_3c0c8ba6eec34be4a6ea569d66038106 [id=4d2f7fabf8a44155a586a2b79a8fca53]). Expected version 1 got 0
I20250114 02:54:38.693131 1206 consensus_queue.cc:260] T ddad7584337345109ce102702008ab37 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:38.693887 468 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 797d6cdf36704c98aa106aac87a945ca (table default.Table_5a9480c6474848d6b400e7dd274d15b0 [id=a17ad30ef32f4cbcb22b5a7d1c7096ff]). Expected version 1 got 0
I20250114 02:54:38.695505 468 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 5d545ac27a7f49faace0655713405cea (table default.Table_b4dc2961391c43fcb08fa062d4e45b3c [id=491b34b03ce14bd9be16e04915468928]). Expected version 1 got 0
I20250114 02:54:38.695776 1206 ts_tablet_manager.cc:1428] T ddad7584337345109ce102702008ab37 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20250114 02:54:38.696799 816 raft_consensus.cc:2463] T 45301f8823ed4ec494e792d41f14f71f P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1.
I20250114 02:54:38.681435 959 raft_consensus.cc:2463] T 45301f8823ed4ec494e792d41f14f71f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 0.
I20250114 02:54:38.693539 958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45301f8823ed4ec494e792d41f14f71f" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
I20250114 02:54:38.697995 958 raft_consensus.cc:3054] T 45301f8823ed4ec494e792d41f14f71f P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:38.697978 1042 leader_election.cc:304] T 45301f8823ed4ec494e792d41f14f71f P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:38.698758 2510 raft_consensus.cc:2798] T 45301f8823ed4ec494e792d41f14f71f P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:38.699558 2510 raft_consensus.cc:695] T 45301f8823ed4ec494e792d41f14f71f P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Becoming Leader. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Running, Role: LEADER
I20250114 02:54:38.700675 466 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet f0e6054998df447287fdff45a6504050
I20250114 02:54:38.700538 2510 consensus_queue.cc:237] T 45301f8823ed4ec494e792d41f14f71f P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:38.702086 466 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:33540 (request call id 321) took 7038 ms. Trace:
I20250114 02:54:38.703541 1205 raft_consensus.cc:738] T 2d4eaf90be6d497d8a3c4d9ac418f4e9 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:38.705071 785 tablet_service.cc:1514] Processing DeleteTablet for tablet f0e6054998df447287fdff45a6504050 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:34 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:38.705440 1079 tablet_service.cc:1514] Processing DeleteTablet for tablet f0e6054998df447287fdff45a6504050 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:34 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:38.702358 466 rpcz_store.cc:276] 0114 02:54:31.663795 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:34.723081 (+3059286us) service_pool.cc:225] Handling call
0114 02:54:34.724313 (+ 1232us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0114 02:54:34.725450 (+ 1137us) catalog_manager.cc:2753] Modifying in-memory table state
0114 02:54:34.725578 (+ 128us) catalog_manager.cc:2762] Locking tablets
0114 02:54:34.740775 (+ 15197us) catalog_manager.cc:2776] Removing table and tablets from system table
0114 02:54:34.880296 (+139521us) catalog_manager.cc:2794] Removing table from by-name map
0114 02:54:34.880507 (+ 211us) catalog_manager.cc:2810] Committing in-memory state
0114 02:54:38.700426 (+3819919us) catalog_manager.cc:2814] Aborting table tasks
0114 02:54:38.702016 (+ 1590us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:34.858777 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:34.859102 (+ 325us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:34.859124 (+ 22us) write_op.cc:435] Acquired schema lock
0114 02:54:34.859136 (+ 12us) tablet.cc:659] Decoding operations
0114 02:54:34.859420 (+ 284us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:34.859452 (+ 32us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:34.859472 (+ 20us) tablet.cc:682] Acquiring locks for 2 operations
0114 02:54:34.859729 (+ 257us) tablet.cc:698] Row locks acquired
0114 02:54:34.859741 (+ 12us) write_op.cc:260] PREPARE: finished
0114 02:54:34.859842 (+ 101us) write_op.cc:270] Start()
0114 02:54:34.859911 (+ 69us) write_op.cc:276] Timestamp: P: 1736823274859783 usec, L: 0
0114 02:54:34.859926 (+ 15us) op_driver.cc:348] REPLICATION: starting
0114 02:54:34.860514 (+ 588us) log.cc:844] Serialized 728 byte log entry
0114 02:54:34.872761 (+ 12247us) op_driver.cc:464] REPLICATION: finished
0114 02:54:34.874582 (+ 1821us) write_op.cc:301] APPLY: starting
0114 02:54:34.874682 (+ 100us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:34.874808 (+ 126us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:34.874909 (+ 101us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:34.875102 (+ 193us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:34.875240 (+ 138us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0114 02:54:34.875281 (+ 41us) write_op.cc:312] APPLY: finished
0114 02:54:34.875440 (+ 159us) log.cc:844] Serialized 29 byte log entry
0114 02:54:34.875547 (+ 107us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:34.875605 (+ 58us) write_op.cc:454] Released schema lock
0114 02:54:34.875647 (+ 42us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:34.903832 (+ 28185us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:34.903854 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1715,"apply.run_cpu_time_us":1223,"apply.run_wall_time_us":1224,"num_ops":2,"prepare.queue_time_us":115929,"prepare.run_cpu_time_us":1900,"prepare.run_wall_time_us":45025,"raft.queue_time_us":791,"raft.run_cpu_time_us":8,"raft.run_wall_time_us":8,"replication_time_us":12773}]]}
W20250114 02:54:38.707877 466 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.66:50663 (request call id 58) took 2142 ms (client timeout 1000 ms). Trace:
W20250114 02:54:38.708310 466 rpcz_store.cc:269] 0114 02:54:36.565044 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:38.707705 (+2142661us) service_pool.cc:212] Skipping call since client already timed out
0114 02:54:38.707808 (+ 103us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250114 02:54:38.709611 466 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.67:34825 (request call id 57) took 1882 ms (client timeout 1000 ms). Trace:
W20250114 02:54:38.710099 466 rpcz_store.cc:269] 0114 02:54:36.827019 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:38.709381 (+1882362us) service_pool.cc:212] Skipping call since client already timed out
0114 02:54:38.709548 (+ 167us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250114 02:54:38.711676 466 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.65:49805 (request call id 62) took 1400 ms (client timeout 1000 ms). Trace:
W20250114 02:54:38.712051 466 rpcz_store.cc:269] 0114 02:54:37.311736 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:38.711366 (+1399630us) service_pool.cc:212] Skipping call since client already timed out
0114 02:54:38.711591 (+ 225us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250114 02:54:38.713567 466 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.0.107.67:34825 (request call id 58) took 881 ms (client timeout 1000 ms). Trace:
W20250114 02:54:38.713845 466 rpcz_store.cc:269] 0114 02:54:37.832148 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:38.713389 (+881241us) service_pool.cc:225] Handling call
0114 02:54:38.713516 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:54:38.715026 1153 heartbeater.cc:346] Connected to a master server at 127.0.107.126:45367
I20250114 02:54:38.718326 1206 tablet_bootstrap.cc:492] T dbb1d0851221497ca2b41844526b4502 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:38.720471 935 tablet_service.cc:1514] Processing DeleteTablet for tablet f0e6054998df447287fdff45a6504050 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:34 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:38.721355 1205 consensus_queue.cc:260] T 2d4eaf90be6d497d8a3c4d9ac418f4e9 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:38.725965 1207 tablet_bootstrap.cc:492] T ab9a85db127b4d82ae1161717ed6006a P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:38.726497 1207 ts_tablet_manager.cc:1397] T ab9a85db127b4d82ae1161717ed6006a P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.127s user 0.010s sys 0.000s
I20250114 02:54:38.733246 1207 raft_consensus.cc:357] T ab9a85db127b4d82ae1161717ed6006a P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:38.733937 1205 ts_tablet_manager.cc:1428] T 2d4eaf90be6d497d8a3c4d9ac418f4e9 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.068s user 0.005s sys 0.000s
I20250114 02:54:38.734058 1207 raft_consensus.cc:383] T ab9a85db127b4d82ae1161717ed6006a P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:38.734397 1207 raft_consensus.cc:738] T ab9a85db127b4d82ae1161717ed6006a P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
W20250114 02:54:38.735277 458 catalog_manager.cc:4852] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet f0e6054998df447287fdff45a6504050 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f0e6054998df447287fdff45a6504050 already in progress: creating tablet
W20250114 02:54:38.738274 458 catalog_manager.cc:4852] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet f0e6054998df447287fdff45a6504050 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f0e6054998df447287fdff45a6504050 already in progress: creating tablet
I20250114 02:54:38.754791 1206 tablet_bootstrap.cc:654] T dbb1d0851221497ca2b41844526b4502 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:38.776561 2623 tablet.cc:1721] T a52d6749ff834b62a8e20a2f45935240 P 525198593c8e47afaee54e74267b87e3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250114 02:54:38.780817 958 raft_consensus.cc:2463] T 45301f8823ed4ec494e792d41f14f71f P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1.
I20250114 02:54:38.762475 2500 raft_consensus.cc:491] T 2d4eaf90be6d497d8a3c4d9ac418f4e9 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:38.781701 2500 raft_consensus.cc:513] T 2d4eaf90be6d497d8a3c4d9ac418f4e9 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:38.783492 2500 leader_election.cc:290] T 2d4eaf90be6d497d8a3c4d9ac418f4e9 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:38.788436 1205 tablet_bootstrap.cc:492] T 46eb13912f754bf1b1fe9d0ad429c3c1 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
W20250114 02:54:38.796648 456 catalog_manager.cc:4852] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet f0e6054998df447287fdff45a6504050 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f0e6054998df447287fdff45a6504050 already in progress: creating tablet
I20250114 02:54:38.804817 465 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet 670dc7ff15d644e18c296eeb2a6b99e1
I20250114 02:54:38.804980 1207 consensus_queue.cc:260] T ab9a85db127b4d82ae1161717ed6006a P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:38.806216 465 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:33540 (request call id 320) took 7146 ms. Trace:
I20250114 02:54:38.809576 785 tablet_service.cc:1514] Processing DeleteTablet for tablet 670dc7ff15d644e18c296eeb2a6b99e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:34 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:38.811017 935 tablet_service.cc:1514] Processing DeleteTablet for tablet 670dc7ff15d644e18c296eeb2a6b99e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:34 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:38.811055 1079 tablet_service.cc:1514] Processing DeleteTablet for tablet 670dc7ff15d644e18c296eeb2a6b99e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:34 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:38.811844 2627 tablet_replica.cc:331] T 670dc7ff15d644e18c296eeb2a6b99e1 P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:38.806511 465 rpcz_store.cc:276] 0114 02:54:31.659799 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:34.634956 (+2975157us) service_pool.cc:225] Handling call
0114 02:54:34.727223 (+ 92267us) spinlock_profiling.cc:243] Waited 25.3 ms on lock 0x7b5400003390. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f645a63f911 00007f645a7bf460 00007f64633d0c26 00007f64633d0673 00007f64634ec64f 00007f6461254b44 00007f6461254ad2 00007f6461254a01 00007f646125497c 00007f6461253c32 00007f645f512dfc
0114 02:54:34.727278 (+ 55us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0114 02:54:34.727876 (+ 598us) catalog_manager.cc:2753] Modifying in-memory table state
0114 02:54:34.727992 (+ 116us) catalog_manager.cc:2762] Locking tablets
0114 02:54:34.728989 (+ 997us) catalog_manager.cc:2776] Removing table and tablets from system table
0114 02:54:34.799253 (+ 70264us) catalog_manager.cc:2794] Removing table from by-name map
0114 02:54:34.851992 (+ 52739us) catalog_manager.cc:2810] Committing in-memory state
0114 02:54:38.804564 (+3952572us) catalog_manager.cc:2814] Aborting table tasks
0114 02:54:38.806161 (+ 1597us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:34.731814 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:34.732118 (+ 304us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:34.732139 (+ 21us) write_op.cc:435] Acquired schema lock
0114 02:54:34.732153 (+ 14us) tablet.cc:659] Decoding operations
0114 02:54:34.732500 (+ 347us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:34.732534 (+ 34us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:34.732556 (+ 22us) tablet.cc:682] Acquiring locks for 2 operations
0114 02:54:34.732801 (+ 245us) tablet.cc:698] Row locks acquired
0114 02:54:34.732813 (+ 12us) write_op.cc:260] PREPARE: finished
0114 02:54:34.732910 (+ 97us) write_op.cc:270] Start()
0114 02:54:34.732986 (+ 76us) write_op.cc:276] Timestamp: P: 1736823274732867 usec, L: 0
0114 02:54:34.733001 (+ 15us) op_driver.cc:348] REPLICATION: starting
0114 02:54:34.733422 (+ 421us) log.cc:844] Serialized 728 byte log entry
0114 02:54:34.744292 (+ 10870us) op_driver.cc:464] REPLICATION: finished
0114 02:54:34.749604 (+ 5312us) write_op.cc:301] APPLY: starting
0114 02:54:34.749658 (+ 54us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:34.749771 (+ 113us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:34.749862 (+ 91us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:34.750112 (+ 250us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:34.750243 (+ 131us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0114 02:54:34.750277 (+ 34us) write_op.cc:312] APPLY: finished
0114 02:54:34.750461 (+ 184us) log.cc:844] Serialized 29 byte log entry
0114 02:54:34.750592 (+ 131us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:34.750671 (+ 79us) write_op.cc:454] Released schema lock
0114 02:54:34.750699 (+ 28us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:34.750892 (+ 193us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:34.750913 (+ 21us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":58261888,"child_traces":[["op",{"apply.queue_time_us":380,"apply.run_cpu_time_us":1290,"apply.run_wall_time_us":1291,"num_ops":2,"prepare.queue_time_us":106,"prepare.run_cpu_time_us":1845,"prepare.run_wall_time_us":1845,"raft.queue_time_us":1113,"raft.run_cpu_time_us":519,"raft.run_wall_time_us":519,"replication_time_us":11153}]]}
I20250114 02:54:38.812717 2627 raft_consensus.cc:2238] T 670dc7ff15d644e18c296eeb2a6b99e1 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:38.813186 2627 raft_consensus.cc:2267] T 670dc7ff15d644e18c296eeb2a6b99e1 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:38.815389 2627 ts_tablet_manager.cc:1905] T 670dc7ff15d644e18c296eeb2a6b99e1 P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:38.816337 1207 ts_tablet_manager.cc:1428] T ab9a85db127b4d82ae1161717ed6006a P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.089s user 0.005s sys 0.000s
I20250114 02:54:38.818140 1205 tablet_bootstrap.cc:654] T 46eb13912f754bf1b1fe9d0ad429c3c1 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:38.837344 471 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:33540 (request call id 327) took 5078 ms. Trace:
I20250114 02:54:38.842959 2532 raft_consensus.cc:491] T ab9a85db127b4d82ae1161717ed6006a P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:38.843417 2532 raft_consensus.cc:513] T ab9a85db127b4d82ae1161717ed6006a P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
W20250114 02:54:38.844871 898 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.67:45749, user_credentials={real_user=slave}} blocked reactor thread for 137817us
I20250114 02:54:38.846583 2532 leader_election.cc:290] T ab9a85db127b4d82ae1161717ed6006a P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:38.850032 2624 tablet_replica.cc:331] T 670dc7ff15d644e18c296eeb2a6b99e1 P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:38.854521 1100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9a85db127b4d82ae1161717ed6006a" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:38.855795 1100 raft_consensus.cc:2463] T ab9a85db127b4d82ae1161717ed6006a P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 0.
I20250114 02:54:38.858461 1079 tablet_service.cc:1514] Processing DeleteTablet for tablet 3161f75bc4e84000af022faa9804c810 with delete_type TABLET_DATA_DELETED (replaced by tablet 53929328fa51408aa98d7825018b6fdb) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:38.858760 753 leader_election.cc:304] T ab9a85db127b4d82ae1161717ed6006a P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:38.860077 2606 raft_consensus.cc:2798] T ab9a85db127b4d82ae1161717ed6006a P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:38.860496 2606 raft_consensus.cc:491] T ab9a85db127b4d82ae1161717ed6006a P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:38.860846 2606 raft_consensus.cc:3054] T ab9a85db127b4d82ae1161717ed6006a P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:38.863104 2484 raft_consensus.cc:491] T ddad7584337345109ce102702008ab37 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:38.863556 2484 raft_consensus.cc:513] T ddad7584337345109ce102702008ab37 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:38.865483 2484 leader_election.cc:290] T ddad7584337345109ce102702008ab37 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:38.870572 935 tablet_service.cc:1514] Processing DeleteTablet for tablet 97f4e061d3b94ee9982410b7ce135453 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:21 UTC) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:38.871693 1076 tablet_service.cc:1514] Processing DeleteTablet for tablet ed7f575653c848d7b076c17c38c4a244 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:26 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:38.872453 933 tablet_service.cc:1514] Processing DeleteTablet for tablet c86562688d5b4fffb71cbdf7be0afc37 with delete_type TABLET_DATA_DELETED (replaced by tablet e7ca0bd1dbd542de9ee11e6c72522ae0) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:38.872684 1076 tablet_service.cc:1514] Processing DeleteTablet for tablet c86562688d5b4fffb71cbdf7be0afc37 with delete_type TABLET_DATA_DELETED (replaced by tablet e7ca0bd1dbd542de9ee11e6c72522ae0) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:38.872988 935 tablet_service.cc:1514] Processing DeleteTablet for tablet 59a46bc865cc48df8e98598764fd1b54 with delete_type TABLET_DATA_DELETED (replaced by tablet cb8aa4212982417b861323c4809017d4) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:38.873070 936 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e858a8449444ef999b9bbef612249d3 with delete_type TABLET_DATA_DELETED (replaced by tablet 797d6cdf36704c98aa106aac87a945ca) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:38.873502 1076 tablet_service.cc:1514] Processing DeleteTablet for tablet 97f4e061d3b94ee9982410b7ce135453 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:21 UTC) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:38.874318 1076 tablet_service.cc:1514] Processing DeleteTablet for tablet 99b940d872ce4781907517c40fe08781 with delete_type TABLET_DATA_DELETED (replaced by tablet fbeddf984d1f466192f60033715855ad) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:38.875700 1079 tablet_service.cc:1514] Processing DeleteTablet for tablet d05089eae2b74f9fb23a226d358a59da with delete_type TABLET_DATA_DELETED (replaced by tablet a5676ce60b6b42ea94fc620be5d4ca03) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:38.875751 1076 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e858a8449444ef999b9bbef612249d3 with delete_type TABLET_DATA_DELETED (replaced by tablet 797d6cdf36704c98aa106aac87a945ca) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:38.876595 1079 tablet_service.cc:1514] Processing DeleteTablet for tablet 59a46bc865cc48df8e98598764fd1b54 with delete_type TABLET_DATA_DELETED (replaced by tablet cb8aa4212982417b861323c4809017d4) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:38.882684 1100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddad7584337345109ce102702008ab37" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:38.837682 471 rpcz_store.cc:276] 0114 02:54:33.759214 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:34.880794 (+1121580us) service_pool.cc:225] Handling call
0114 02:54:34.970108 (+ 89314us) spinlock_profiling.cc:243] Waited 88.5 ms on lock 0x7b54000316d4. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616cad50 00007f646174df2a 00007f64633f2dec 00007f64634ec986 00007f6461259644 00007f64612595d2 00007f6461259501 00007f646125947c 00007f6461258732 00007f645f512dfc 00007f645f512236
0114 02:54:34.984581 (+ 14473us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0114 02:54:34.985211 (+ 630us) catalog_manager.cc:3627] Apply alter schema
0114 02:54:34.985757 (+ 546us) catalog_manager.cc:3659] Acquired catalog manager lock
0114 02:54:34.985955 (+ 198us) catalog_manager.cc:3817] Updating metadata on disk
0114 02:54:35.132941 (+146986us) catalog_manager.cc:3848] Committing alterations to in-memory state
0114 02:54:38.837176 (+3704235us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:35.087688 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:35.097396 (+ 9708us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:35.097418 (+ 22us) write_op.cc:435] Acquired schema lock
0114 02:54:35.097438 (+ 20us) tablet.cc:659] Decoding operations
0114 02:54:35.097721 (+ 283us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:35.097761 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:35.097782 (+ 21us) tablet.cc:682] Acquiring locks for 1 operations
0114 02:54:35.097911 (+ 129us) tablet.cc:698] Row locks acquired
0114 02:54:35.097924 (+ 13us) write_op.cc:260] PREPARE: finished
0114 02:54:35.098026 (+ 102us) write_op.cc:270] Start()
0114 02:54:35.098096 (+ 70us) write_op.cc:276] Timestamp: P: 1736823275097983 usec, L: 0
0114 02:54:35.098111 (+ 15us) op_driver.cc:348] REPLICATION: starting
0114 02:54:35.098541 (+ 430us) log.cc:844] Serialized 405 byte log entry
0114 02:54:35.122785 (+ 24244us) op_driver.cc:464] REPLICATION: finished
0114 02:54:35.124375 (+ 1590us) write_op.cc:301] APPLY: starting
0114 02:54:35.124424 (+ 49us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:35.124504 (+ 80us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:35.124594 (+ 90us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:35.124716 (+ 122us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:35.124833 (+ 117us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:35.124869 (+ 36us) write_op.cc:312] APPLY: finished
0114 02:54:35.125029 (+ 160us) log.cc:844] Serialized 23 byte log entry
0114 02:54:35.125145 (+ 116us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:35.125197 (+ 52us) write_op.cc:454] Released schema lock
0114 02:54:35.125224 (+ 27us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:35.125427 (+ 203us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:35.125455 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":3762,"spinlock_wait_cycles":203531136,"child_traces":[["op",{"apply.queue_time_us":1451,"apply.run_cpu_time_us":1027,"apply.run_wall_time_us":1028,"num_ops":1,"prepare.queue_time_us":100706,"prepare.run_cpu_time_us":1694,"prepare.run_wall_time_us":11019,"replication_time_us":24600}]]}
W20250114 02:54:38.885285 753 leader_election.cc:343] T ddad7584337345109ce102702008ab37 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:38.885347 471 master_service.cc:946] ReplaceTablet: received request to replace tablet 232d6a697f744e179a1a8da492a7d227 from {username='slave'} at 127.0.0.1:33540
I20250114 02:54:38.886461 471 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:33540 (request call id 334) took 3288 ms. Trace:
I20250114 02:54:38.886747 471 rpcz_store.cc:276] 0114 02:54:35.598417 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:38.885220 (+3286803us) service_pool.cc:225] Handling call
0114 02:54:38.886397 (+ 1177us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:54:38.886484 2606 raft_consensus.cc:513] T ab9a85db127b4d82ae1161717ed6006a P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:38.887585 471 master_service.cc:946] ReplaceTablet: received request to replace tablet ab9a85db127b4d82ae1161717ed6006a from {username='slave'} at 127.0.0.1:33540
I20250114 02:54:38.892318 429 master-stress-test.cc:579] Killed test thread 6/10
I20250114 02:54:38.892565 429 master-stress-test.cc:577] Killing test thread 7/10
I20250114 02:54:38.893127 429 master-stress-test.cc:579] Killed test thread 7/10
I20250114 02:54:38.893322 429 master-stress-test.cc:577] Killing test thread 8/10
I20250114 02:54:38.893572 429 master-stress-test.cc:579] Killed test thread 8/10
I20250114 02:54:38.893747 429 master-stress-test.cc:577] Killing test thread 9/10
I20250114 02:54:38.895843 429 master-stress-test.cc:579] Killed test thread 9/10
I20250114 02:54:38.896051 429 master-stress-test.cc:577] Killing test thread 10/10
I20250114 02:54:38.896014 936 tablet_service.cc:1514] Processing DeleteTablet for tablet d05089eae2b74f9fb23a226d358a59da with delete_type TABLET_DATA_DELETED (replaced by tablet a5676ce60b6b42ea94fc620be5d4ca03) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:38.896309 935 tablet_service.cc:1514] Processing DeleteTablet for tablet ed7f575653c848d7b076c17c38c4a244 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:26 UTC) from {username='slave'} at 127.0.0.1:55068
W20250114 02:54:38.897341 470 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.66:50663 (request call id 56) took 4978 ms (client timeout 1000 ms). Trace:
W20250114 02:54:38.899639 467 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.67:34825 (request call id 55) took 4662 ms (client timeout 1000 ms). Trace:
W20250114 02:54:38.899942 467 rpcz_store.cc:269] 0114 02:54:34.237541 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:34.523271 (+285730us) service_pool.cc:225] Handling call
0114 02:54:35.378873 (+855602us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:35.378898 (+ 25us) write_op.cc:454] Released schema lock
0114 02:54:38.899481 (+3520583us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:35.290296 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:35.290640 (+ 344us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:35.290662 (+ 22us) write_op.cc:435] Acquired schema lock
0114 02:54:35.290676 (+ 14us) tablet.cc:659] Decoding operations
0114 02:54:35.290913 (+ 237us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:35.290953 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:35.290973 (+ 20us) tablet.cc:682] Acquiring locks for 1 operations
0114 02:54:35.291109 (+ 136us) tablet.cc:698] Row locks acquired
0114 02:54:35.291122 (+ 13us) write_op.cc:260] PREPARE: finished
0114 02:54:35.298301 (+ 7179us) write_op.cc:270] Start()
0114 02:54:35.298413 (+ 112us) write_op.cc:276] Timestamp: P: 1736823275298216 usec, L: 0
0114 02:54:35.298429 (+ 16us) op_driver.cc:348] REPLICATION: starting
0114 02:54:35.298935 (+ 506us) log.cc:844] Serialized 530 byte log entry
0114 02:54:35.310171 (+ 11236us) op_driver.cc:464] REPLICATION: finished
0114 02:54:35.310566 (+ 395us) write_op.cc:301] APPLY: starting
0114 02:54:35.310621 (+ 55us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:35.310709 (+ 88us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:35.310810 (+ 101us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:35.310953 (+ 143us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:35.311080 (+ 127us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:35.311118 (+ 38us) write_op.cc:312] APPLY: finished
0114 02:54:35.311284 (+ 166us) log.cc:844] Serialized 23 byte log entry
0114 02:54:35.311439 (+ 155us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:35.311500 (+ 61us) write_op.cc:454] Released schema lock
0114 02:54:35.311528 (+ 28us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":46151,"reported_tablets":39,"child_traces":[["op",{"apply.queue_time_us":242,"apply.run_cpu_time_us":1129,"apply.run_wall_time_us":1129,"num_ops":1,"prepare.queue_time_us":127,"prepare.run_cpu_time_us":1939,"prepare.run_wall_time_us":34219,"raft.queue_time_us":135,"raft.run_cpu_time_us":601,"raft.run_wall_time_us":1381,"replication_time_us":11632}]]}
I20250114 02:54:38.841805 462 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: default.Table_0a0bda55a4a9476c98d961822b8a19bd [id=fb2be5c1ca9444a197f3315ea9c59b09] alter complete (version 1)
I20250114 02:54:38.912853 2484 consensus_queue.cc:1035] T 97de018391f84b0ebd3130861681b7b5 P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:54:38.914809 1205 tablet_bootstrap.cc:492] T 46eb13912f754bf1b1fe9d0ad429c3c1 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:38.915186 1205 ts_tablet_manager.cc:1397] T 46eb13912f754bf1b1fe9d0ad429c3c1 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.139s user 0.009s sys 0.002s
W20250114 02:54:38.897634 470 rpcz_store.cc:269] 0114 02:54:33.918716 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:34.436514 (+517798us) service_pool.cc:225] Handling call
0114 02:54:34.730961 (+294447us) spinlock_profiling.cc:243] Waited 82.8 ms on lock 0x7b5400003390. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f645a63f911 00007f645a7bf460 00007f64633e7cb6 00007f64634eb6ae 00007f6461248fc4 00007f6461248f52 00007f6461248e81 00007f6461248dfc 00007f64612480b2 00007f645f512dfc 00007f645f512236
0114 02:54:38.897271 (+4166310us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:36.327863 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:36.330413 (+ 2550us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:36.330438 (+ 25us) write_op.cc:435] Acquired schema lock
0114 02:54:36.330451 (+ 13us) tablet.cc:659] Decoding operations
0114 02:54:36.330884 (+ 433us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:36.330924 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:36.330946 (+ 22us) tablet.cc:682] Acquiring locks for 3 operations
0114 02:54:36.331187 (+ 241us) tablet.cc:698] Row locks acquired
0114 02:54:36.331200 (+ 13us) write_op.cc:260] PREPARE: finished
0114 02:54:36.331310 (+ 110us) write_op.cc:270] Start()
0114 02:54:36.331391 (+ 81us) write_op.cc:276] Timestamp: P: 1736823276331249 usec, L: 0
0114 02:54:36.331404 (+ 13us) op_driver.cc:348] REPLICATION: starting
0114 02:54:36.331914 (+ 510us) log.cc:844] Serialized 1284 byte log entry
0114 02:54:36.341870 (+ 9956us) op_driver.cc:464] REPLICATION: finished
0114 02:54:36.342178 (+ 308us) write_op.cc:301] APPLY: starting
0114 02:54:36.342228 (+ 50us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:36.342360 (+ 132us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:36.342473 (+ 113us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:36.342937 (+ 464us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:36.343089 (+ 152us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0114 02:54:36.343130 (+ 41us) write_op.cc:312] APPLY: finished
0114 02:54:36.343355 (+ 225us) log.cc:844] Serialized 35 byte log entry
0114 02:54:36.343505 (+ 150us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:36.343574 (+ 69us) write_op.cc:454] Released schema lock
0114 02:54:36.343602 (+ 28us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:36.343796 (+ 194us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:36.343817 (+ 21us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":258,"reported_tablets":36,"spinlock_wait_cycles":190501120,"child_traces":[["op",{"apply.queue_time_us":177,"apply.run_cpu_time_us":1589,"apply.run_wall_time_us":1589,"num_ops":3,"prepare.queue_time_us":80006,"prepare.run_cpu_time_us":2157,"prepare.run_wall_time_us":4310,"raft.queue_time_us":1202,"raft.run_cpu_time_us":622,"raft.run_wall_time_us":622,"replication_time_us":10379}]]}
I20250114 02:54:38.941655 2627 ts_tablet_manager.cc:1918] T 670dc7ff15d644e18c296eeb2a6b99e1 P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:38.941699 2624 raft_consensus.cc:2238] T 670dc7ff15d644e18c296eeb2a6b99e1 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:38.942212 2624 raft_consensus.cc:2267] T 670dc7ff15d644e18c296eeb2a6b99e1 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:38.948576 2627 log.cc:1198] T 670dc7ff15d644e18c296eeb2a6b99e1 P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/670dc7ff15d644e18c296eeb2a6b99e1
I20250114 02:54:38.949498 2624 ts_tablet_manager.cc:1905] T 670dc7ff15d644e18c296eeb2a6b99e1 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:38.950176 2627 ts_tablet_manager.cc:1939] T 670dc7ff15d644e18c296eeb2a6b99e1 P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:38.958570 2638 tablet.cc:1721] T a52d6749ff834b62a8e20a2f45935240 P 819f9e3412e54d44abb11d8b3ca877c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250114 02:54:38.965302 1205 raft_consensus.cc:357] T 46eb13912f754bf1b1fe9d0ad429c3c1 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:38.966043 1205 raft_consensus.cc:383] T 46eb13912f754bf1b1fe9d0ad429c3c1 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:38.966311 1205 raft_consensus.cc:738] T 46eb13912f754bf1b1fe9d0ad429c3c1 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:38.960347 811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d4eaf90be6d497d8a3c4d9ac418f4e9" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:38.966737 811 raft_consensus.cc:2463] T 2d4eaf90be6d497d8a3c4d9ac418f4e9 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:38.966920 1205 consensus_queue.cc:260] T 46eb13912f754bf1b1fe9d0ad429c3c1 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:38.975721 1103 raft_consensus.cc:3054] T 797d6cdf36704c98aa106aac87a945ca P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:38.977428 1100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d4eaf90be6d497d8a3c4d9ac418f4e9" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
W20250114 02:54:38.983315 469 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.67:34825 (request call id 56) took 3660 ms (client timeout 1000 ms). Trace:
W20250114 02:54:38.983765 469 rpcz_store.cc:269] 0114 02:54:35.322671 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:35.342366 (+ 19695us) service_pool.cc:225] Handling call
0114 02:54:35.471687 (+129321us) spinlock_profiling.cc:243] Waited 123 ms on lock 0x7b54000316d4. stack: 00007f645a7c9f63 00007f645a7c9e60 00007f645a340584 00000000004fcd24 00000000004fccb9 00007f64616cad50 00007f646174df2a 00007f64633f2dec 00007f64634eadee 00007f6461248fc4 00007f6461248f52 00007f6461248e81 00007f6461248dfc 00007f64612480b2 00007f645f512dfc 00007f645f512236
0114 02:54:38.569403 (+3097716us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:38.569426 (+ 23us) write_op.cc:454] Released schema lock
0114 02:54:38.983215 (+413789us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:38.550081 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:38.550515 (+ 434us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:38.550539 (+ 24us) write_op.cc:435] Acquired schema lock
0114 02:54:38.551078 (+ 539us) tablet.cc:659] Decoding operations
0114 02:54:38.551342 (+ 264us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:38.551378 (+ 36us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:38.551402 (+ 24us) tablet.cc:682] Acquiring locks for 1 operations
0114 02:54:38.551555 (+ 153us) tablet.cc:698] Row locks acquired
0114 02:54:38.551578 (+ 23us) write_op.cc:260] PREPARE: finished
0114 02:54:38.551726 (+ 148us) write_op.cc:270] Start()
0114 02:54:38.551831 (+ 105us) write_op.cc:276] Timestamp: P: 1736823278551550 usec, L: 0
0114 02:54:38.551855 (+ 24us) op_driver.cc:348] REPLICATION: starting
0114 02:54:38.552528 (+ 673us) log.cc:844] Serialized 530 byte log entry
0114 02:54:38.562343 (+ 9815us) op_driver.cc:464] REPLICATION: finished
0114 02:54:38.562783 (+ 440us) write_op.cc:301] APPLY: starting
0114 02:54:38.562846 (+ 63us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:38.562929 (+ 83us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:38.563022 (+ 93us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:38.563177 (+ 155us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:38.563299 (+ 122us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:38.563336 (+ 37us) write_op.cc:312] APPLY: finished
0114 02:54:38.563508 (+ 172us) log.cc:844] Serialized 23 byte log entry
0114 02:54:38.563636 (+ 128us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:38.563694 (+ 58us) write_op.cc:454] Released schema lock
0114 02:54:38.563726 (+ 32us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":66430,"reported_tablets":42,"spinlock_wait_cycles":283412352,"child_traces":[["op",{"apply.queue_time_us":302,"apply.run_cpu_time_us":1103,"apply.run_wall_time_us":1102,"num_ops":1,"prepare.queue_time_us":1398,"prepare.run_cpu_time_us":2783,"prepare.run_wall_time_us":2782,"raft.queue_time_us":1306,"raft.run_cpu_time_us":622,"raft.run_wall_time_us":622,"replication_time_us":10397}]]}
I20250114 02:54:38.998687 2406 raft_consensus.cc:491] T 038ea1ea231e490ba27ecd5430883058 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:38.999119 2406 raft_consensus.cc:513] T 038ea1ea231e490ba27ecd5430883058 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:39.001608 2406 leader_election.cc:290] T 038ea1ea231e490ba27ecd5430883058 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:39.029057 2640 tablet.cc:1721] T a52d6749ff834b62a8e20a2f45935240 P 90d6c2c58bb54e0eafb365625ac33871: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250114 02:54:39.041997 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet 670dc7ff15d644e18c296eeb2a6b99e1 (table default.Table_fd43493446ca49a48e8a970bb92ea239 [id=6dbd2db73b5443a2af5d3b5056ab0169]) successfully deleted
I20250114 02:54:39.042769 2627 tablet_replica.cc:331] T 97f4e061d3b94ee9982410b7ce135453 P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:39.043644 959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "038ea1ea231e490ba27ecd5430883058" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:39.045493 2606 leader_election.cc:290] T ab9a85db127b4d82ae1161717ed6006a P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Requested vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:39.051623 1206 tablet_bootstrap.cc:492] T dbb1d0851221497ca2b41844526b4502 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:39.051532 957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d4eaf90be6d497d8a3c4d9ac418f4e9" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:39.052008 1206 ts_tablet_manager.cc:1397] T dbb1d0851221497ca2b41844526b4502 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.334s user 0.021s sys 0.035s
I20250114 02:54:39.052093 957 raft_consensus.cc:2463] T 2d4eaf90be6d497d8a3c4d9ac418f4e9 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 0.
I20250114 02:54:39.054067 1206 raft_consensus.cc:357] T dbb1d0851221497ca2b41844526b4502 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:39.052459 1205 ts_tablet_manager.cc:1428] T 46eb13912f754bf1b1fe9d0ad429c3c1 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.137s user 0.006s sys 0.001s
I20250114 02:54:39.054733 1206 raft_consensus.cc:383] T dbb1d0851221497ca2b41844526b4502 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:39.052850 959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46eb13912f754bf1b1fe9d0ad429c3c1" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:39.054958 1206 raft_consensus.cc:738] T dbb1d0851221497ca2b41844526b4502 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:39.055246 959 raft_consensus.cc:2463] T 46eb13912f754bf1b1fe9d0ad429c3c1 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 0.
I20250114 02:54:39.055955 1206 consensus_queue.cc:260] T dbb1d0851221497ca2b41844526b4502 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:39.061478 1205 tablet_bootstrap.cc:492] T 6d741bfd00784d2d830be3ebe50dc7c4 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:39.064632 960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2c3f3a89b3d47f1b0dfacfac8218805" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:39.074045 1100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9a85db127b4d82ae1161717ed6006a" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:39.074604 1100 raft_consensus.cc:3054] T ab9a85db127b4d82ae1161717ed6006a P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:39.077750 960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9a85db127b4d82ae1161717ed6006a" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:39.079421 960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddad7584337345109ce102702008ab37" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:39.082693 960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9a85db127b4d82ae1161717ed6006a" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
I20250114 02:54:39.093466 1104 raft_consensus.cc:3054] T 038ea1ea231e490ba27ecd5430883058 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:39.092386 2642 raft_consensus.cc:491] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b303b7df086340a0b5ad6373940ae750)
I20250114 02:54:39.096275 1205 tablet_bootstrap.cc:654] T 6d741bfd00784d2d830be3ebe50dc7c4 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:39.103750 2642 raft_consensus.cc:513] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b303b7df086340a0b5ad6373940ae750" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 45367 } } peers { permanent_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } } peers { permanent_uuid: "87b3db09ac87436e9a60d6de5fedaff5" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } }
I20250114 02:54:39.105989 1205 tablet_bootstrap.cc:492] T 6d741bfd00784d2d830be3ebe50dc7c4 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:39.106349 1205 ts_tablet_manager.cc:1397] T 6d741bfd00784d2d830be3ebe50dc7c4 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.045s user 0.008s sys 0.000s
I20250114 02:54:39.109789 898 leader_election.cc:304] T 2d4eaf90be6d497d8a3c4d9ac418f4e9 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
I20250114 02:54:39.110424 2500 raft_consensus.cc:2798] T 2d4eaf90be6d497d8a3c4d9ac418f4e9 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:39.110741 2500 raft_consensus.cc:491] T 2d4eaf90be6d497d8a3c4d9ac418f4e9 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:39.111032 2500 raft_consensus.cc:3054] T 2d4eaf90be6d497d8a3c4d9ac418f4e9 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:39.112478 1205 raft_consensus.cc:357] T 6d741bfd00784d2d830be3ebe50dc7c4 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:39.115849 1205 raft_consensus.cc:383] T 6d741bfd00784d2d830be3ebe50dc7c4 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:39.118425 2625 tablet_replica.cc:331] T 670dc7ff15d644e18c296eeb2a6b99e1 P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:39.119340 2625 raft_consensus.cc:2238] T 670dc7ff15d644e18c296eeb2a6b99e1 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Raft consensus shutting down.
I20250114 02:54:39.120206 2625 raft_consensus.cc:2267] T 670dc7ff15d644e18c296eeb2a6b99e1 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:39.121138 1207 tablet_bootstrap.cc:492] T 550e84ac4d1b4ef58dfdea6e552ee940 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:39.122257 2625 ts_tablet_manager.cc:1905] T 670dc7ff15d644e18c296eeb2a6b99e1 P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:39.122972 1100 raft_consensus.cc:2463] T ab9a85db127b4d82ae1161717ed6006a P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 1.
I20250114 02:54:39.128962 1207 tablet_bootstrap.cc:654] T 550e84ac4d1b4ef58dfdea6e552ee940 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:39.119961 2500 raft_consensus.cc:513] T 2d4eaf90be6d497d8a3c4d9ac418f4e9 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:39.133782 2623 tablet.cc:1721] T fc406eab968745d893f1a8538ff61f37 P 525198593c8e47afaee54e74267b87e3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250114 02:54:39.144896 1205 raft_consensus.cc:738] T 6d741bfd00784d2d830be3ebe50dc7c4 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:39.135694 2627 raft_consensus.cc:2238] T 97f4e061d3b94ee9982410b7ce135453 P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Raft consensus shutting down.
I20250114 02:54:39.145766 1205 consensus_queue.cc:260] T 6d741bfd00784d2d830be3ebe50dc7c4 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:39.146406 2627 raft_consensus.cc:2267] T 97f4e061d3b94ee9982410b7ce135453 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:39.147814 1205 ts_tablet_manager.cc:1428] T 6d741bfd00784d2d830be3ebe50dc7c4 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.041s user 0.005s sys 0.000s
I20250114 02:54:39.148628 2627 ts_tablet_manager.cc:1905] T 97f4e061d3b94ee9982410b7ce135453 P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:39.149566 2642 leader_election.cc:290] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b303b7df086340a0b5ad6373940ae750 (127.0.107.126:45367), 87b3db09ac87436e9a60d6de5fedaff5 (127.0.107.124:39105)
W20250114 02:54:39.165768 898 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.65:39397, user_credentials={real_user=slave}} blocked reactor thread for 54845.7us
I20250114 02:54:39.164077 2500 leader_election.cc:290] T 2d4eaf90be6d497d8a3c4d9ac418f4e9 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Requested vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:39.123937 753 leader_election.cc:304] T ab9a85db127b4d82ae1161717ed6006a P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2, 90d6c2c58bb54e0eafb365625ac33871; no voters:
W20250114 02:54:39.189803 463 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.65:49805 (request call id 59) took 5261 ms (client timeout 1000 ms). Trace:
I20250114 02:54:39.196416 1205 tablet_bootstrap.cc:492] T 550e84ac4d1b4ef58dfdea6e552ee940 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:39.201542 785 tablet_service.cc:1514] Processing DeleteTablet for tablet 97f4e061d3b94ee9982410b7ce135453 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:21 UTC) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:39.202033 784 tablet_service.cc:1514] Processing DeleteTablet for tablet 3161f75bc4e84000af022faa9804c810 with delete_type TABLET_DATA_DELETED (replaced by tablet 53929328fa51408aa98d7825018b6fdb) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:39.202379 2624 ts_tablet_manager.cc:1918] T 670dc7ff15d644e18c296eeb2a6b99e1 P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:39.200711 2627 ts_tablet_manager.cc:1918] T 97f4e061d3b94ee9982410b7ce135453 P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:39.202737 2624 log.cc:1198] T 670dc7ff15d644e18c296eeb2a6b99e1 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/670dc7ff15d644e18c296eeb2a6b99e1
I20250114 02:54:39.202860 2627 log.cc:1198] T 97f4e061d3b94ee9982410b7ce135453 P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/97f4e061d3b94ee9982410b7ce135453
I20250114 02:54:39.203470 2624 ts_tablet_manager.cc:1939] T 670dc7ff15d644e18c296eeb2a6b99e1 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:39.203522 2627 ts_tablet_manager.cc:1939] T 97f4e061d3b94ee9982410b7ce135453 P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:39.205240 786 tablet_service.cc:1514] Processing DeleteTablet for tablet d05089eae2b74f9fb23a226d358a59da with delete_type TABLET_DATA_DELETED (replaced by tablet a5676ce60b6b42ea94fc620be5d4ca03) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:39.205626 787 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b8361cd538046e0bcb4aac821748f67 with delete_type TABLET_DATA_DELETED (replaced by tablet f0e6054998df447287fdff45a6504050) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:39.206215 797 tablet_service.cc:1514] Processing DeleteTablet for tablet d56255ca806d4e2cac09fbb5e1b1efbd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-14 02:54:27 UTC) from {username='slave'} at 127.0.0.1:53078
W20250114 02:54:39.206823 753 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.67:45749, user_credentials={real_user=slave}} blocked reactor thread for 82958.1us
I20250114 02:54:39.207681 2636 raft_consensus.cc:2798] T ab9a85db127b4d82ae1161717ed6006a P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:39.207813 1207 tablet_bootstrap.cc:492] T 550e84ac4d1b4ef58dfdea6e552ee940 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:39.208187 1207 ts_tablet_manager.cc:1397] T 550e84ac4d1b4ef58dfdea6e552ee940 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.087s user 0.011s sys 0.000s
I20250114 02:54:39.208122 784 tablet_service.cc:1514] Processing DeleteTablet for tablet 99b940d872ce4781907517c40fe08781 with delete_type TABLET_DATA_DELETED (replaced by tablet fbeddf984d1f466192f60033715855ad) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:39.208122 2636 raft_consensus.cc:695] T ab9a85db127b4d82ae1161717ed6006a P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 LEADER]: Becoming Leader. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Running, Role: LEADER
W20250114 02:54:39.210839 898 leader_election.cc:343] T 2d4eaf90be6d497d8a3c4d9ac418f4e9 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:39.212827 1207 raft_consensus.cc:357] T 550e84ac4d1b4ef58dfdea6e552ee940 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:39.213585 1207 raft_consensus.cc:383] T 550e84ac4d1b4ef58dfdea6e552ee940 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:39.213819 1207 raft_consensus.cc:738] T 550e84ac4d1b4ef58dfdea6e552ee940 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:39.216936 2624 tablet_replica.cc:331] T 0b8361cd538046e0bcb4aac821748f67 P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
W20250114 02:54:39.219911 751 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.66:44037, user_credentials={real_user=slave}} blocked reactor thread for 319725us
I20250114 02:54:39.210779 2636 consensus_queue.cc:237] T ab9a85db127b4d82ae1161717ed6006a P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:39.226701 813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "038ea1ea231e490ba27ecd5430883058" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:39.227409 813 raft_consensus.cc:2388] T 038ea1ea231e490ba27ecd5430883058 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 90d6c2c58bb54e0eafb365625ac33871 in current term 1: Already voted for candidate 525198593c8e47afaee54e74267b87e3 in this term.
I20250114 02:54:39.229076 2447 consensus_queue.cc:1035] T 797d6cdf36704c98aa106aac87a945ca P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:39.230767 1206 ts_tablet_manager.cc:1428] T dbb1d0851221497ca2b41844526b4502 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.178s user 0.004s sys 0.002s
I20250114 02:54:39.231384 2624 raft_consensus.cc:2238] T 0b8361cd538046e0bcb4aac821748f67 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:39.231621 1206 tablet_bootstrap.cc:492] T 992bb16197b74be5b1dc884f29d9c240 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:39.231884 2624 raft_consensus.cc:2267] T 0b8361cd538046e0bcb4aac821748f67 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:39.233371 2627 tablet_replica.cc:331] T 59a46bc865cc48df8e98598764fd1b54 P 525198593c8e47afaee54e74267b87e3: stopping tablet replica
I20250114 02:54:39.234227 2627 raft_consensus.cc:2238] T 59a46bc865cc48df8e98598764fd1b54 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:39.234587 2624 ts_tablet_manager.cc:1905] T 0b8361cd538046e0bcb4aac821748f67 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:39.234704 2627 raft_consensus.cc:2267] T 59a46bc865cc48df8e98598764fd1b54 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:39.234172 1207 consensus_queue.cc:260] T 550e84ac4d1b4ef58dfdea6e552ee940 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:39.235285 2622 consensus_queue.cc:1035] T 038ea1ea231e490ba27ecd5430883058 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:39.240370 818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d4eaf90be6d497d8a3c4d9ac418f4e9" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2"
I20250114 02:54:39.240509 2638 tablet.cc:1721] T fc406eab968745d893f1a8538ff61f37 P 819f9e3412e54d44abb11d8b3ca877c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250114 02:54:39.240890 818 raft_consensus.cc:3054] T 2d4eaf90be6d497d8a3c4d9ac418f4e9 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:39.248973 818 raft_consensus.cc:2463] T 2d4eaf90be6d497d8a3c4d9ac418f4e9 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 1.
I20250114 02:54:39.254462 751 leader_election.cc:304] T 2d4eaf90be6d497d8a3c4d9ac418f4e9 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters: 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:39.256330 2484 raft_consensus.cc:2758] T 2d4eaf90be6d497d8a3c4d9ac418f4e9 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20250114 02:54:39.257422 751 leader_election.cc:304] T 46eb13912f754bf1b1fe9d0ad429c3c1 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters: 90d6c2c58bb54e0eafb365625ac33871
W20250114 02:54:39.275126 2322 debug-util.cc:398] Leaking SignalData structure 0x7b08000ccdc0 after lost signal to thread 1487
I20250114 02:54:39.274761 2484 raft_consensus.cc:2798] T 46eb13912f754bf1b1fe9d0ad429c3c1 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:39.277702 2484 raft_consensus.cc:491] T 46eb13912f754bf1b1fe9d0ad429c3c1 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:39.278126 2484 raft_consensus.cc:3054] T 46eb13912f754bf1b1fe9d0ad429c3c1 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:39.264220 2627 ts_tablet_manager.cc:1905] T 59a46bc865cc48df8e98598764fd1b54 P 525198593c8e47afaee54e74267b87e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:39.286173 2484 raft_consensus.cc:513] T 46eb13912f754bf1b1fe9d0ad429c3c1 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:39.290412 2447 consensus_queue.cc:1035] T 797d6cdf36704c98aa106aac87a945ca P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:39.294588 2447 consensus_queue.cc:1035] T 038ea1ea231e490ba27ecd5430883058 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250114 02:54:39.276350 751 leader_election.cc:343] T a2c3f3a89b3d47f1b0dfacfac8218805 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:39.295957 898 leader_election.cc:304] T 2d4eaf90be6d497d8a3c4d9ac418f4e9 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
I20250114 02:54:39.296504 751 leader_election.cc:304] T a2c3f3a89b3d47f1b0dfacfac8218805 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:39.297353 2447 raft_consensus.cc:2798] T 2d4eaf90be6d497d8a3c4d9ac418f4e9 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election won for term 1
W20250114 02:54:39.297947 751 leader_election.cc:343] T ab9a85db127b4d82ae1161717ed6006a P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:39.298096 2634 raft_consensus.cc:2743] T a2c3f3a89b3d47f1b0dfacfac8218805 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:39.274495 1206 tablet_bootstrap.cc:654] T 992bb16197b74be5b1dc884f29d9c240 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:39.298763 2447 raft_consensus.cc:695] T 2d4eaf90be6d497d8a3c4d9ac418f4e9 P 525198593c8e47afaee54e74267b87e3 [term 1 LEADER]: Becoming Leader. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Running, Role: LEADER
I20250114 02:54:39.299446 2447 consensus_queue.cc:237] T 2d4eaf90be6d497d8a3c4d9ac418f4e9 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
W20250114 02:54:39.300542 751 leader_election.cc:343] T ddad7584337345109ce102702008ab37 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:39.302970 751 leader_election.cc:304] T ddad7584337345109ce102702008ab37 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:39.305820 1205 tablet_bootstrap.cc:654] T 550e84ac4d1b4ef58dfdea6e552ee940 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
W20250114 02:54:39.307901 751 leader_election.cc:343] T ab9a85db127b4d82ae1161717ed6006a P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:39.308574 2484 leader_election.cc:290] T 46eb13912f754bf1b1fe9d0ad429c3c1 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Requested vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:39.310108 2635 raft_consensus.cc:2743] T ddad7584337345109ce102702008ab37 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:39.321754 2606 raft_consensus.cc:491] T dbb1d0851221497ca2b41844526b4502 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:39.322508 1206 tablet_bootstrap.cc:492] T 992bb16197b74be5b1dc884f29d9c240 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:39.322247 2606 raft_consensus.cc:513] T dbb1d0851221497ca2b41844526b4502 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:39.322939 1206 ts_tablet_manager.cc:1397] T 992bb16197b74be5b1dc884f29d9c240 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.091s user 0.006s sys 0.004s
I20250114 02:54:39.324321 2606 leader_election.cc:290] T dbb1d0851221497ca2b41844526b4502 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:39.324483 2510 raft_consensus.cc:491] T 797d6cdf36704c98aa106aac87a945ca P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:39.324898 2510 raft_consensus.cc:513] T 797d6cdf36704c98aa106aac87a945ca P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:39.325116 1206 raft_consensus.cc:357] T 992bb16197b74be5b1dc884f29d9c240 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:39.325734 1206 raft_consensus.cc:383] T 992bb16197b74be5b1dc884f29d9c240 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:39.326006 1206 raft_consensus.cc:738] T 992bb16197b74be5b1dc884f29d9c240 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:39.341334 1206 consensus_queue.cc:260] T 992bb16197b74be5b1dc884f29d9c240 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
W20250114 02:54:39.344394 1040 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.66:44037, user_credentials={real_user=slave}} blocked reactor thread for 60765.7us
I20250114 02:54:39.348287 2640 tablet.cc:1721] T d2a7b2bee5c1440fb705b172c18a9247 P 90d6c2c58bb54e0eafb365625ac33871: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250114 02:54:39.344067 1206 ts_tablet_manager.cc:1428] T 992bb16197b74be5b1dc884f29d9c240 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.021s user 0.005s sys 0.000s
I20250114 02:54:39.191793 481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" candidate_term: 2 candidate_status { last_received { term: 1 index: 439 } } ignore_live_leader: false dest_uuid: "b303b7df086340a0b5ad6373940ae750" is_pre_election: true
I20250114 02:54:39.352434 1206 tablet_bootstrap.cc:492] T 5535411d3a27445e99a051b501fe8b39 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
W20250114 02:54:39.352972 463 rpcz_store.cc:269] 0114 02:54:33.928325 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:34.472327 (+544002us) service_pool.cc:225] Handling call
0114 02:54:39.189730 (+4717403us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":37446,"reported_tablets":41}
W20250114 02:54:39.197197 462 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.65:49805 (request call id 60) took 4079 ms (client timeout 1000 ms). Trace:
I20250114 02:54:39.217471 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet 670dc7ff15d644e18c296eeb2a6b99e1 (table default.Table_fd43493446ca49a48e8a970bb92ea239 [id=6dbd2db73b5443a2af5d3b5056ab0169]) successfully deleted
W20250114 02:54:39.356469 458 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.107.65:39397, user_credentials={real_user=slave}} blocked reactor thread for 139018us
W20250114 02:54:39.354679 462 rpcz_store.cc:269] 0114 02:54:35.117705 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:35.128118 (+ 10413us) service_pool.cc:225] Handling call
0114 02:54:39.197137 (+4069019us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:38.139058 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:38.139462 (+ 404us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:38.139514 (+ 52us) write_op.cc:435] Acquired schema lock
0114 02:54:38.139529 (+ 15us) tablet.cc:659] Decoding operations
0114 02:54:38.139798 (+ 269us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:38.139846 (+ 48us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:38.139874 (+ 28us) tablet.cc:682] Acquiring locks for 1 operations
0114 02:54:38.140018 (+ 144us) tablet.cc:698] Row locks acquired
0114 02:54:38.140034 (+ 16us) write_op.cc:260] PREPARE: finished
0114 02:54:38.147324 (+ 7290us) write_op.cc:270] Start()
0114 02:54:38.147420 (+ 96us) write_op.cc:276] Timestamp: P: 1736823278140065 usec, L: 0
0114 02:54:38.147438 (+ 18us) op_driver.cc:348] REPLICATION: starting
0114 02:54:38.149355 (+ 1917us) log.cc:844] Serialized 356 byte log entry
0114 02:54:38.177307 (+ 27952us) op_driver.cc:464] REPLICATION: finished
0114 02:54:38.177637 (+ 330us) write_op.cc:301] APPLY: starting
0114 02:54:38.177686 (+ 49us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:38.177768 (+ 82us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:38.177863 (+ 95us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:38.178035 (+ 172us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:38.178148 (+ 113us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:38.178204 (+ 56us) write_op.cc:312] APPLY: finished
0114 02:54:38.178360 (+ 156us) log.cc:844] Serialized 23 byte log entry
0114 02:54:38.178493 (+ 133us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:38.178551 (+ 58us) write_op.cc:454] Released schema lock
0114 02:54:38.178580 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0114 02:54:38.178787 (+ 207us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:38.178810 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":89647,"reported_tablets":43,"child_traces":[["op",{"apply.queue_time_us":212,"apply.run_cpu_time_us":1108,"apply.run_wall_time_us":1108,"num_ops":1,"prepare.queue_time_us":21876,"prepare.run_cpu_time_us":2224,"prepare.run_wall_time_us":21401,"raft.queue_time_us":695,"raft.run_cpu_time_us":636,"raft.run_wall_time_us":636,"replication_time_us":29769,"thread_start_us":22870,"threads_started":1}]]}
W20250114 02:54:39.357203 458 catalog_manager.cc:4839] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet 97f4e061d3b94ee9982410b7ce135453 because the tablet was not found. No further retry: Not found: Tablet not found: 97f4e061d3b94ee9982410b7ce135453
W20250114 02:54:39.357903 458 catalog_manager.cc:4839] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet 3161f75bc4e84000af022faa9804c810 because the tablet was not found. No further retry: Not found: Tablet not found: 3161f75bc4e84000af022faa9804c810
I20250114 02:54:39.357967 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" candidate_term: 2 candidate_status { last_received { term: 1 index: 439 } } ignore_live_leader: false dest_uuid: "87b3db09ac87436e9a60d6de5fedaff5" is_pre_election: true
W20250114 02:54:39.358613 458 catalog_manager.cc:4839] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): delete failed for tablet d05089eae2b74f9fb23a226d358a59da because the tablet was not found. No further retry: Not found: Tablet not found: d05089eae2b74f9fb23a226d358a59da
I20250114 02:54:39.227742 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet 97f4e061d3b94ee9982410b7ce135453 (table default.Table_6a4fe39298514187b65def5841d3a992 [id=5c5a7974ad2f4e58ba5c47a6eeaaac62]) successfully deleted
I20250114 02:54:39.358893 2294 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 687d4ff6d3cb42cf90a965070b0d8a70 in term 1.
W20250114 02:54:39.359462 456 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.107.66:44037, user_credentials={real_user=slave}} blocked reactor thread for 131716us
I20250114 02:54:39.359861 2510 leader_election.cc:290] T 797d6cdf36704c98aa106aac87a945ca P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:39.360554 530 leader_election.cc:304] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 687d4ff6d3cb42cf90a965070b0d8a70, 87b3db09ac87436e9a60d6de5fedaff5; no voters: b303b7df086340a0b5ad6373940ae750
W20250114 02:54:39.361070 456 catalog_manager.cc:4839] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet c86562688d5b4fffb71cbdf7be0afc37 because the tablet was not found. No further retry: Not found: Tablet not found: c86562688d5b4fffb71cbdf7be0afc37
I20250114 02:54:39.362159 1100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d4eaf90be6d497d8a3c4d9ac418f4e9" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:39.364746 2642 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:54:39.370249 1104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46eb13912f754bf1b1fe9d0ad429c3c1" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:39.336995 1207 ts_tablet_manager.cc:1428] T 550e84ac4d1b4ef58dfdea6e552ee940 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.128s user 0.006s sys 0.000s
I20250114 02:54:39.371702 1207 tablet_bootstrap.cc:492] T 46eb13912f754bf1b1fe9d0ad429c3c1 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:39.385310 1206 tablet_bootstrap.cc:654] T 5535411d3a27445e99a051b501fe8b39 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:39.399288 1103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbb1d0851221497ca2b41844526b4502" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:39.400439 2642 raft_consensus.cc:491] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b303b7df086340a0b5ad6373940ae750)
I20250114 02:54:39.400853 2642 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [term 1 FOLLOWER]: Advancing to term 2
W20250114 02:54:39.415516 753 leader_election.cc:343] T 46eb13912f754bf1b1fe9d0ad429c3c1 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
W20250114 02:54:39.311889 464 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.66:50663 (request call id 57) took 4013 ms (client timeout 1000 ms). Trace:
I20250114 02:54:39.420436 962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "797d6cdf36704c98aa106aac87a945ca" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:39.420816 2623 tablet.cc:1721] T d2a7b2bee5c1440fb705b172c18a9247 P 525198593c8e47afaee54e74267b87e3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250114 02:54:39.426333 753 leader_election.cc:343] T dbb1d0851221497ca2b41844526b4502 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:39.434193 813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "797d6cdf36704c98aa106aac87a945ca" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:39.434801 957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbb1d0851221497ca2b41844526b4502" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
W20250114 02:54:39.437060 751 leader_election.cc:343] T dbb1d0851221497ca2b41844526b4502 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:39.437453 751 leader_election.cc:304] T dbb1d0851221497ca2b41844526b4502 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 819f9e3412e54d44abb11d8b3ca877c2; no voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:39.438198 2640 tablet.cc:1721] T fc406eab968745d893f1a8538ff61f37 P 90d6c2c58bb54e0eafb365625ac33871: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250114 02:54:39.438596 2606 raft_consensus.cc:2743] T dbb1d0851221497ca2b41844526b4502 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:39.451417 1206 tablet_bootstrap.cc:492] T 5535411d3a27445e99a051b501fe8b39 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:39.451829 1206 ts_tablet_manager.cc:1397] T 5535411d3a27445e99a051b501fe8b39 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.100s user 0.007s sys 0.004s
I20250114 02:54:39.454442 2627 ts_tablet_manager.cc:1918] T 59a46bc865cc48df8e98598764fd1b54 P 525198593c8e47afaee54e74267b87e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:39.455133 2627 log.cc:1198] T 59a46bc865cc48df8e98598764fd1b54 P 525198593c8e47afaee54e74267b87e3: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/59a46bc865cc48df8e98598764fd1b54
W20250114 02:54:39.460912 898 leader_election.cc:343] T 2d4eaf90be6d497d8a3c4d9ac418f4e9 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:39.461584 958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46eb13912f754bf1b1fe9d0ad429c3c1" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
I20250114 02:54:39.462193 958 raft_consensus.cc:3054] T 46eb13912f754bf1b1fe9d0ad429c3c1 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:39.475836 2642 raft_consensus.cc:513] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b303b7df086340a0b5ad6373940ae750" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 45367 } } peers { permanent_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } } peers { permanent_uuid: "87b3db09ac87436e9a60d6de5fedaff5" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } }
I20250114 02:54:39.477000 556 raft_consensus.cc:1235] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [term 2 FOLLOWER]: Rejecting Update request from peer b303b7df086340a0b5ad6373940ae750 for earlier term 1. Current term is 2. Ops: [1.441-1.441]
I20250114 02:54:39.477672 2627 ts_tablet_manager.cc:1939] T 59a46bc865cc48df8e98598764fd1b54 P 525198593c8e47afaee54e74267b87e3: Deleting consensus metadata
I20250114 02:54:39.485966 2662 consensus_queue.cc:1046] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 }, Status: INVALID_TERM, Last received: 1.440, Next index: 441, Last known committed idx: 439, Time since last communication: 0.001s
I20250114 02:54:39.486855 2662 raft_consensus.cc:3049] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 02:54:39.487192 2662 raft_consensus.cc:738] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b303b7df086340a0b5ad6373940ae750, State: Running, Role: LEADER
I20250114 02:54:39.501886 1042 leader_election.cc:304] T 038ea1ea231e490ba27ecd5430883058 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90d6c2c58bb54e0eafb365625ac33871; no voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2
I20250114 02:54:39.503193 2406 raft_consensus.cc:2743] T 038ea1ea231e490ba27ecd5430883058 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:39.454907 1206 raft_consensus.cc:357] T 5535411d3a27445e99a051b501fe8b39 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:39.504194 1206 raft_consensus.cc:383] T 5535411d3a27445e99a051b501fe8b39 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:39.504455 1206 raft_consensus.cc:738] T 5535411d3a27445e99a051b501fe8b39 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:39.505028 1206 consensus_queue.cc:260] T 5535411d3a27445e99a051b501fe8b39 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:39.507660 1206 ts_tablet_manager.cc:1428] T 5535411d3a27445e99a051b501fe8b39 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.056s user 0.000s sys 0.005s
I20250114 02:54:39.513378 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" candidate_term: 2 candidate_status { last_received { term: 1 index: 440 } } ignore_live_leader: false dest_uuid: "87b3db09ac87436e9a60d6de5fedaff5"
I20250114 02:54:39.513938 2294 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:54:39.522759 2294 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 687d4ff6d3cb42cf90a965070b0d8a70 in term 2.
I20250114 02:54:39.523820 530 leader_election.cc:304] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 687d4ff6d3cb42cf90a965070b0d8a70, 87b3db09ac87436e9a60d6de5fedaff5; no voters:
I20250114 02:54:39.524518 1207 tablet_bootstrap.cc:654] T 46eb13912f754bf1b1fe9d0ad429c3c1 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:39.534229 2662 consensus_queue.cc:260] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 440, Committed index: 440, Last appended: 1.441, Last appended by leader: 441, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b303b7df086340a0b5ad6373940ae750" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 45367 } } peers { permanent_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } } peers { permanent_uuid: "87b3db09ac87436e9a60d6de5fedaff5" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } }
I20250114 02:54:39.539077 2638 tablet.cc:1721] T df045d80d8ce4ff194a6d2039a44bce6 P 819f9e3412e54d44abb11d8b3ca877c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250114 02:54:39.540877 2662 raft_consensus.cc:3054] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750 [term 1 FOLLOWER]: Advancing to term 2
I20250114 02:54:39.541623 2642 leader_election.cc:290] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [CANDIDATE]: Term 2 election: Requested vote from peers b303b7df086340a0b5ad6373940ae750 (127.0.107.126:45367), 87b3db09ac87436e9a60d6de5fedaff5 (127.0.107.124:39105)
I20250114 02:54:39.553732 2642 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 02:54:39.554592 1042 leader_election.cc:304] T 797d6cdf36704c98aa106aac87a945ca P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90d6c2c58bb54e0eafb365625ac33871; no voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2
I20250114 02:54:39.555828 2525 raft_consensus.cc:2743] T 797d6cdf36704c98aa106aac87a945ca P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:54:39.559885 2625 ts_tablet_manager.cc:1918] T 670dc7ff15d644e18c296eeb2a6b99e1 P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:39.560482 2642 raft_consensus.cc:695] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [term 2 LEADER]: Becoming Leader. State: Replica: 687d4ff6d3cb42cf90a965070b0d8a70, State: Running, Role: LEADER
I20250114 02:54:39.559643 2640 tablet.cc:1721] T df045d80d8ce4ff194a6d2039a44bce6 P 90d6c2c58bb54e0eafb365625ac33871: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250114 02:54:39.568465 2625 log.cc:1198] T 670dc7ff15d644e18c296eeb2a6b99e1 P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/670dc7ff15d644e18c296eeb2a6b99e1
I20250114 02:54:39.569423 2625 ts_tablet_manager.cc:1939] T 670dc7ff15d644e18c296eeb2a6b99e1 P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
W20250114 02:54:39.417943 464 rpcz_store.cc:269] 0114 02:54:35.298953 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:35.299081 (+ 128us) service_pool.cc:225] Handling call
0114 02:54:38.423743 (+3124662us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:38.423766 (+ 23us) write_op.cc:454] Released schema lock
0114 02:54:39.311825 (+888059us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:38.403113 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:38.403534 (+ 421us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:38.403563 (+ 29us) write_op.cc:435] Acquired schema lock
0114 02:54:38.403578 (+ 15us) tablet.cc:659] Decoding operations
0114 02:54:38.403914 (+ 336us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:38.403961 (+ 47us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:38.403984 (+ 23us) tablet.cc:682] Acquiring locks for 2 operations
0114 02:54:38.404269 (+ 285us) tablet.cc:698] Row locks acquired
0114 02:54:38.404288 (+ 19us) write_op.cc:260] PREPARE: finished
0114 02:54:38.404417 (+ 129us) write_op.cc:270] Start()
0114 02:54:38.404511 (+ 94us) write_op.cc:276] Timestamp: P: 1736823278404253 usec, L: 0
0114 02:54:38.404525 (+ 14us) op_driver.cc:348] REPLICATION: starting
0114 02:54:38.405001 (+ 476us) log.cc:844] Serialized 907 byte log entry
0114 02:54:38.416493 (+ 11492us) op_driver.cc:464] REPLICATION: finished
0114 02:54:38.416790 (+ 297us) write_op.cc:301] APPLY: starting
0114 02:54:38.416835 (+ 45us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:38.416928 (+ 93us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:38.417016 (+ 88us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:38.417268 (+ 252us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:38.417393 (+ 125us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0114 02:54:38.417430 (+ 37us) write_op.cc:312] APPLY: finished
0114 02:54:38.417601 (+ 171us) log.cc:844] Serialized 29 byte log entry
0114 02:54:38.417735 (+ 134us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:38.417813 (+ 78us) write_op.cc:454] Released schema lock
0114 02:54:38.417850 (+ 37us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":15702,"reported_tablets":39,"child_traces":[["op",{"apply.queue_time_us":166,"apply.run_cpu_time_us":1230,"apply.run_wall_time_us":1229,"num_ops":2,"prepare.queue_time_us":7853,"prepare.run_cpu_time_us":2152,"prepare.run_wall_time_us":2155,"raft.queue_time_us":1602,"raft.run_cpu_time_us":719,"raft.run_wall_time_us":720,"replication_time_us":11868}]]}
W20250114 02:54:39.570466 530 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.124:39105, user_credentials={real_user=slave}} blocked reactor thread for 46747.1us
I20250114 02:54:39.572894 1206 tablet_bootstrap.cc:492] T 9ffec1d1bb594fda83df8d94d4533d56 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:39.569175 2642 consensus_queue.cc:237] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 439, Committed index: 439, Last appended: 1.440, Last appended by leader: 440, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b303b7df086340a0b5ad6373940ae750" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 45367 } } peers { permanent_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } } peers { permanent_uuid: "87b3db09ac87436e9a60d6de5fedaff5" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } }
I20250114 02:54:39.581532 2671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 687d4ff6d3cb42cf90a965070b0d8a70. Latest consensus state: current_term: 2 leader_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b303b7df086340a0b5ad6373940ae750" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 45367 } } peers { permanent_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } } peers { permanent_uuid: "87b3db09ac87436e9a60d6de5fedaff5" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } } }
I20250114 02:54:39.582450 2671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [sys.catalog]: This master's current role is: LEADER
W20250114 02:54:39.596844 898 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.67:45749, user_credentials={real_user=slave}} blocked reactor thread for 96875.9us
I20250114 02:54:39.606698 1206 tablet_bootstrap.cc:654] T 9ffec1d1bb594fda83df8d94d4533d56 P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:39.607977 456 catalog_manager.cc:4872] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): tablet 59a46bc865cc48df8e98598764fd1b54 (table default.Table_1097f09fb10e441fb639f8d7aa12d487 [id=d8c9691818d042d7b49475fa009c4ca1]) successfully deleted
W20250114 02:54:39.609102 456 catalog_manager.cc:4839] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet 6e858a8449444ef999b9bbef612249d3 because the tablet was not found. No further retry: Not found: Tablet not found: 6e858a8449444ef999b9bbef612249d3
W20250114 02:54:39.610968 456 catalog_manager.cc:4839] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet d05089eae2b74f9fb23a226d358a59da because the tablet was not found. No further retry: Not found: Tablet not found: d05089eae2b74f9fb23a226d358a59da
I20250114 02:54:39.611444 2638 tablet.cc:1721] T d2a7b2bee5c1440fb705b172c18a9247 P 819f9e3412e54d44abb11d8b3ca877c2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250114 02:54:39.629961 2636 raft_consensus.cc:491] T 5535411d3a27445e99a051b501fe8b39 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:39.630468 2636 raft_consensus.cc:513] T 5535411d3a27445e99a051b501fe8b39 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:39.634012 1206 tablet_bootstrap.cc:492] T 9ffec1d1bb594fda83df8d94d4533d56 P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:39.634392 1206 ts_tablet_manager.cc:1397] T 9ffec1d1bb594fda83df8d94d4533d56 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.062s user 0.008s sys 0.001s
I20250114 02:54:39.635078 2636 leader_election.cc:290] T 5535411d3a27445e99a051b501fe8b39 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:39.636624 1206 raft_consensus.cc:357] T 9ffec1d1bb594fda83df8d94d4533d56 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:39.637301 1206 raft_consensus.cc:383] T 9ffec1d1bb594fda83df8d94d4533d56 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:39.637543 1206 raft_consensus.cc:738] T 9ffec1d1bb594fda83df8d94d4533d56 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:39.638948 958 raft_consensus.cc:2463] T 46eb13912f754bf1b1fe9d0ad429c3c1 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 819f9e3412e54d44abb11d8b3ca877c2 in term 1.
I20250114 02:54:39.639225 1206 consensus_queue.cc:260] T 9ffec1d1bb594fda83df8d94d4533d56 P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:39.641283 1206 ts_tablet_manager.cc:1428] T 9ffec1d1bb594fda83df8d94d4533d56 P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250114 02:54:39.642267 1205 tablet_bootstrap.cc:492] T 550e84ac4d1b4ef58dfdea6e552ee940 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:39.642648 1205 ts_tablet_manager.cc:1397] T 550e84ac4d1b4ef58dfdea6e552ee940 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.446s user 0.007s sys 0.014s
I20250114 02:54:39.645056 1205 raft_consensus.cc:357] T 550e84ac4d1b4ef58dfdea6e552ee940 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:39.645648 1205 raft_consensus.cc:383] T 550e84ac4d1b4ef58dfdea6e552ee940 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:39.645889 1205 raft_consensus.cc:738] T 550e84ac4d1b4ef58dfdea6e552ee940 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:39.646510 1205 consensus_queue.cc:260] T 550e84ac4d1b4ef58dfdea6e552ee940 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:39.649827 1207 tablet_bootstrap.cc:492] T 46eb13912f754bf1b1fe9d0ad429c3c1 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:39.650233 1207 ts_tablet_manager.cc:1397] T 46eb13912f754bf1b1fe9d0ad429c3c1 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.279s user 0.008s sys 0.000s
I20250114 02:54:39.652513 1100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5535411d3a27445e99a051b501fe8b39" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
W20250114 02:54:39.653599 753 leader_election.cc:343] T 5535411d3a27445e99a051b501fe8b39 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:39.655714 2625 tablet_replica.cc:331] T 3161f75bc4e84000af022faa9804c810 P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:39.655730 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet 670dc7ff15d644e18c296eeb2a6b99e1 (table default.Table_fd43493446ca49a48e8a970bb92ea239 [id=6dbd2db73b5443a2af5d3b5056ab0169]) successfully deleted
I20250114 02:54:39.662957 1206 tablet_bootstrap.cc:492] T e285f86c29654b9697427516e99ca43a P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
I20250114 02:54:39.668627 481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" candidate_term: 2 candidate_status { last_received { term: 1 index: 440 } } ignore_live_leader: false dest_uuid: "b303b7df086340a0b5ad6373940ae750"
I20250114 02:54:39.669663 481 raft_consensus.cc:2405] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 687d4ff6d3cb42cf90a965070b0d8a70 for term 2 because replica has last-logged OpId of term: 1 index: 441, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 440.
I20250114 02:54:39.674440 2624 ts_tablet_manager.cc:1918] T 0b8361cd538046e0bcb4aac821748f67 P 819f9e3412e54d44abb11d8b3ca877c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:39.674849 2624 log.cc:1198] T 0b8361cd538046e0bcb4aac821748f67 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/0b8361cd538046e0bcb4aac821748f67
I20250114 02:54:39.683691 2513 consensus_queue.cc:1035] T 45301f8823ed4ec494e792d41f14f71f P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:54:39.684305 471 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Sending DeleteTablet for 3 replicas of tablet ab9a85db127b4d82ae1161717ed6006a
I20250114 02:54:39.684617 1207 raft_consensus.cc:357] T 46eb13912f754bf1b1fe9d0ad429c3c1 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:39.685218 1207 raft_consensus.cc:383] T 46eb13912f754bf1b1fe9d0ad429c3c1 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:39.685436 1207 raft_consensus.cc:738] T 46eb13912f754bf1b1fe9d0ad429c3c1 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:39.686057 1207 consensus_queue.cc:260] T 46eb13912f754bf1b1fe9d0ad429c3c1 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:39.686800 784 tablet_service.cc:1514] Processing DeleteTablet for tablet ab9a85db127b4d82ae1161717ed6006a with delete_type TABLET_DATA_DELETED (replaced by tablet f442d63314c54667a3645b5f958c520b) from {username='slave'} at 127.0.0.1:53078
I20250114 02:54:39.687538 1207 ts_tablet_manager.cc:1428] T 46eb13912f754bf1b1fe9d0ad429c3c1 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.037s user 0.005s sys 0.000s
I20250114 02:54:39.688061 471 catalog_manager.cc:6596] ReplaceTablet: tablet ab9a85db127b4d82ae1161717ed6006a deleted and replaced by tablet f442d63314c54667a3645b5f958c520b
I20250114 02:54:39.688521 471 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:33540 (request call id 335) took 4089 ms. Trace:
I20250114 02:54:39.688793 1079 tablet_service.cc:1514] Processing DeleteTablet for tablet ab9a85db127b4d82ae1161717ed6006a with delete_type TABLET_DATA_DELETED (replaced by tablet f442d63314c54667a3645b5f958c520b) from {username='slave'} at 127.0.0.1:53876
I20250114 02:54:39.689164 2624 ts_tablet_manager.cc:1939] T 0b8361cd538046e0bcb4aac821748f67 P 819f9e3412e54d44abb11d8b3ca877c2: Deleting consensus metadata
I20250114 02:54:39.688845 471 rpcz_store.cc:276] 0114 02:54:35.598859 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:38.887481 (+3288622us) service_pool.cc:225] Handling call
0114 02:54:39.688461 (+800980us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:38.918575 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:38.919100 (+ 525us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:38.919135 (+ 35us) write_op.cc:435] Acquired schema lock
0114 02:54:38.919154 (+ 19us) tablet.cc:659] Decoding operations
0114 02:54:38.919553 (+ 399us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:38.919604 (+ 51us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:38.919636 (+ 32us) tablet.cc:682] Acquiring locks for 2 operations
0114 02:54:38.919882 (+ 246us) tablet.cc:698] Row locks acquired
0114 02:54:38.919897 (+ 15us) write_op.cc:260] PREPARE: finished
0114 02:54:38.920017 (+ 120us) write_op.cc:270] Start()
0114 02:54:38.920097 (+ 80us) write_op.cc:276] Timestamp: P: 1736823278919953 usec, L: 0
0114 02:54:38.920113 (+ 16us) op_driver.cc:348] REPLICATION: starting
0114 02:54:38.920763 (+ 650us) log.cc:844] Serialized 625 byte log entry
0114 02:54:39.439800 (+519037us) op_driver.cc:464] REPLICATION: finished
0114 02:54:39.468834 (+ 29034us) write_op.cc:301] APPLY: starting
0114 02:54:39.468887 (+ 53us) tablet.cc:1364] starting BulkCheckPresence
0114 02:54:39.468989 (+ 102us) tablet.cc:1367] finished BulkCheckPresence
0114 02:54:39.469091 (+ 102us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 02:54:39.469423 (+ 332us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 02:54:39.469616 (+ 193us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0114 02:54:39.469653 (+ 37us) write_op.cc:312] APPLY: finished
0114 02:54:39.469838 (+ 185us) log.cc:844] Serialized 29 byte log entry
0114 02:54:39.469977 (+ 139us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:39.470038 (+ 61us) write_op.cc:454] Released schema lock
0114 02:54:39.470068 (+ 30us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":28873,"apply.run_cpu_time_us":1438,"apply.run_wall_time_us":1438,"num_ops":2,"prepare.queue_time_us":93,"prepare.run_cpu_time_us":2520,"prepare.run_wall_time_us":2520,"raft.queue_time_us":1195,"raft.run_cpu_time_us":531,"raft.run_wall_time_us":529,"replication_time_us":519576,"thread_start_us":30400,"threads_started":1}]]}
I20250114 02:54:39.698940 935 tablet_service.cc:1514] Processing DeleteTablet for tablet ab9a85db127b4d82ae1161717ed6006a with delete_type TABLET_DATA_DELETED (replaced by tablet f442d63314c54667a3645b5f958c520b) from {username='slave'} at 127.0.0.1:55068
I20250114 02:54:39.700083 2675 raft_consensus.cc:491] T 6d741bfd00784d2d830be3ebe50dc7c4 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250114 02:54:39.700330 898 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.65:39397, user_credentials={real_user=slave}} blocked reactor thread for 96869.3us
I20250114 02:54:39.700526 2675 raft_consensus.cc:513] T 6d741bfd00784d2d830be3ebe50dc7c4 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:39.702241 2675 leader_election.cc:290] T 6d741bfd00784d2d830be3ebe50dc7c4 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
W20250114 02:54:39.705127 1040 consensus_peers.cc:487] T 59a46bc865cc48df8e98598764fd1b54 P 90d6c2c58bb54e0eafb365625ac33871 -> Peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Couldn't send request to peer 525198593c8e47afaee54e74267b87e3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 59a46bc865cc48df8e98598764fd1b54. This is attempt 1: this message will repeat every 5th retry.
I20250114 02:54:39.706445 1206 tablet_bootstrap.cc:654] T e285f86c29654b9697427516e99ca43a P 819f9e3412e54d44abb11d8b3ca877c2: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:39.709031 1205 ts_tablet_manager.cc:1428] T 550e84ac4d1b4ef58dfdea6e552ee940 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.066s user 0.006s sys 0.001s
I20250114 02:54:39.711411 2624 tablet_replica.cc:331] T d56255ca806d4e2cac09fbb5e1b1efbd P 819f9e3412e54d44abb11d8b3ca877c2: stopping tablet replica
I20250114 02:54:39.712016 751 leader_election.cc:304] T 46eb13912f754bf1b1fe9d0ad429c3c1 P 819f9e3412e54d44abb11d8b3ca877c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters: 90d6c2c58bb54e0eafb365625ac33871
I20250114 02:54:39.713220 2634 raft_consensus.cc:2798] T 46eb13912f754bf1b1fe9d0ad429c3c1 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:39.713616 2634 raft_consensus.cc:695] T 46eb13912f754bf1b1fe9d0ad429c3c1 P 819f9e3412e54d44abb11d8b3ca877c2 [term 1 LEADER]: Becoming Leader. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Running, Role: LEADER
I20250114 02:54:39.714305 2634 consensus_queue.cc:237] T 46eb13912f754bf1b1fe9d0ad429c3c1 P 819f9e3412e54d44abb11d8b3ca877c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:39.725802 2625 raft_consensus.cc:2238] T 3161f75bc4e84000af022faa9804c810 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:39.726423 2625 raft_consensus.cc:2267] T 3161f75bc4e84000af022faa9804c810 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:39.729161 2625 ts_tablet_manager.cc:1905] T 3161f75bc4e84000af022faa9804c810 P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:39.729689 1205 tablet_bootstrap.cc:492] T a2c3f3a89b3d47f1b0dfacfac8218805 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:39.733901 1206 tablet_bootstrap.cc:492] T e285f86c29654b9697427516e99ca43a P 819f9e3412e54d44abb11d8b3ca877c2: No bootstrap required, opened a new log
I20250114 02:54:39.734357 1206 ts_tablet_manager.cc:1397] T e285f86c29654b9697427516e99ca43a P 819f9e3412e54d44abb11d8b3ca877c2: Time spent bootstrapping tablet: real 0.072s user 0.004s sys 0.005s
W20250114 02:54:39.738736 1042 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.65:39397, user_credentials={real_user=slave}} blocked reactor thread for 73303.5us
I20250114 02:54:39.739495 1206 raft_consensus.cc:357] T e285f86c29654b9697427516e99ca43a P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:39.740192 1206 raft_consensus.cc:383] T e285f86c29654b9697427516e99ca43a P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:39.740478 1206 raft_consensus.cc:738] T e285f86c29654b9697427516e99ca43a P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819f9e3412e54d44abb11d8b3ca877c2, State: Initialized, Role: FOLLOWER
I20250114 02:54:39.744500 1206 consensus_queue.cc:260] T e285f86c29654b9697427516e99ca43a P 819f9e3412e54d44abb11d8b3ca877c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } }
I20250114 02:54:39.745180 1207 tablet_bootstrap.cc:492] T 2d4eaf90be6d497d8a3c4d9ac418f4e9 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:39.746021 814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d741bfd00784d2d830be3ebe50dc7c4" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "819f9e3412e54d44abb11d8b3ca877c2" is_pre_election: true
I20250114 02:54:39.746572 814 raft_consensus.cc:2463] T 6d741bfd00784d2d830be3ebe50dc7c4 P 819f9e3412e54d44abb11d8b3ca877c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525198593c8e47afaee54e74267b87e3 in term 0.
I20250114 02:54:39.746057 1206 ts_tablet_manager.cc:1428] T e285f86c29654b9697427516e99ca43a P 819f9e3412e54d44abb11d8b3ca877c2: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20250114 02:54:39.757738 1206 tablet_bootstrap.cc:492] T 53983c5af2444e419ccdd0be395bffa9 P 819f9e3412e54d44abb11d8b3ca877c2: Bootstrap starting.
W20250114 02:54:39.697974 456 catalog_manager.cc:4839] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet ed7f575653c848d7b076c17c38c4a244 because the tablet was not found. No further retry: Not found: Tablet not found: ed7f575653c848d7b076c17c38c4a244
W20250114 02:54:39.760483 456 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.107.66:44037, user_credentials={real_user=slave}} blocked reactor thread for 62532.4us
W20250114 02:54:39.761161 456 catalog_manager.cc:4852] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): delete failed for tablet ab9a85db127b4d82ae1161717ed6006a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ab9a85db127b4d82ae1161717ed6006a already in progress: creating tablet
W20250114 02:54:39.699259 458 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 6027bc3f9121401fbdced24aabaa2d94 (table default.Table_bbdaf710a91a429f9dd60f183ceea9f9 [id=4507c028bd074fd0a1d0231ea671b355], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 6027bc3f9121401fbdced24aabaa2d94 consensus state has no leader
I20250114 02:54:39.761888 458 catalog_manager.cc:4872] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): tablet 0b8361cd538046e0bcb4aac821748f67 (table default.Table_2850a159f211445988d540f3a56ec551 [id=441702f585ce4600be174d46b8607686]) successfully deleted
I20250114 02:54:39.765981 429 master-stress-test.cc:579] Killed test thread 10/10
I20250114 02:54:39.766376 429 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrCUXNq/build/tsan/bin/kudu with pid 717
I20250114 02:54:39.771503 960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5535411d3a27445e99a051b501fe8b39" candidate_uuid: "819f9e3412e54d44abb11d8b3ca877c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:39.773483 2623 tablet.cc:1721] T df045d80d8ce4ff194a6d2039a44bce6 P 525198593c8e47afaee54e74267b87e3: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250114 02:54:39.782280 1102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d741bfd00784d2d830be3ebe50dc7c4" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871" is_pre_election: true
I20250114 02:54:39.791538 465 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 141cede72b8741a8b0615d7c6bb823bb (table default.Table_4364d1df16d149d2ba10469ce7288d4e [id=13fc996d911b430290e37a94dd44b1ab]). Expected version 1 got 0
I20250114 02:54:39.796787 1207 tablet_bootstrap.cc:654] T 2d4eaf90be6d497d8a3c4d9ac418f4e9 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:39.802739 1205 tablet_bootstrap.cc:654] T a2c3f3a89b3d47f1b0dfacfac8218805 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:39.810132 2525 raft_consensus.cc:491] T 550e84ac4d1b4ef58dfdea6e552ee940 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:39.810642 2525 raft_consensus.cc:513] T 550e84ac4d1b4ef58dfdea6e552ee940 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:39.812927 2525 leader_election.cc:290] T 550e84ac4d1b4ef58dfdea6e552ee940 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:39.815640 960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550e84ac4d1b4ef58dfdea6e552ee940" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:39.816166 960 raft_consensus.cc:2463] T 550e84ac4d1b4ef58dfdea6e552ee940 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 0.
I20250114 02:54:39.817281 1040 leader_election.cc:304] T 550e84ac4d1b4ef58dfdea6e552ee940 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:39.818042 898 leader_election.cc:304] T 6d741bfd00784d2d830be3ebe50dc7c4 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2; no voters:
I20250114 02:54:39.818682 2500 raft_consensus.cc:2798] T 6d741bfd00784d2d830be3ebe50dc7c4 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:39.818989 2500 raft_consensus.cc:491] T 6d741bfd00784d2d830be3ebe50dc7c4 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:39.819288 2500 raft_consensus.cc:3054] T 6d741bfd00784d2d830be3ebe50dc7c4 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:39.820511 2510 raft_consensus.cc:2798] T 550e84ac4d1b4ef58dfdea6e552ee940 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:54:39.820835 2510 raft_consensus.cc:491] T 550e84ac4d1b4ef58dfdea6e552ee940 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:54:39.821137 2510 raft_consensus.cc:3054] T 550e84ac4d1b4ef58dfdea6e552ee940 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:39.827582 2500 raft_consensus.cc:513] T 6d741bfd00784d2d830be3ebe50dc7c4 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:39.829742 2500 leader_election.cc:290] T 6d741bfd00784d2d830be3ebe50dc7c4 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Requested vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749)
I20250114 02:54:39.835120 2510 raft_consensus.cc:513] T 550e84ac4d1b4ef58dfdea6e552ee940 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:39.796952 465 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 30a56fb2d941470590e17f37b90aef26 (table default.Table_0e0710e3f43e4e859ad32662704ee41c [id=66082998119e4c43a666fb43087fc622]). Expected version 1 got 0
I20250114 02:54:39.844718 465 catalog_manager.cc:5526] T be7aad41ccdd4ef9bcfe8a985155708f P 819f9e3412e54d44abb11d8b3ca877c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67). New cstate: current_term: 1 leader_uuid: "90d6c2c58bb54e0eafb365625ac33871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } }
I20250114 02:54:39.862849 1099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d741bfd00784d2d830be3ebe50dc7c4" candidate_uuid: "525198593c8e47afaee54e74267b87e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d6c2c58bb54e0eafb365625ac33871"
I20250114 02:54:39.875901 2510 leader_election.cc:290] T 550e84ac4d1b4ef58dfdea6e552ee940 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Requested vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
I20250114 02:54:39.891932 465 catalog_manager.cc:5526] T 97de018391f84b0ebd3130861681b7b5 P 819f9e3412e54d44abb11d8b3ca877c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65). New cstate: current_term: 1 leader_uuid: "819f9e3412e54d44abb11d8b3ca877c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:39.897760 1207 tablet_bootstrap.cc:492] T 2d4eaf90be6d497d8a3c4d9ac418f4e9 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:39.898196 1207 ts_tablet_manager.cc:1397] T 2d4eaf90be6d497d8a3c4d9ac418f4e9 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.210s user 0.013s sys 0.000s
I20250114 02:54:39.900439 465 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet a52d6749ff834b62a8e20a2f45935240 (table default.Table_1c9b7896941e4289b1c12e20c210c2c1 [id=b3ee15b9f10f4d489edbde3af9034d25]). Expected version 1 got 0
I20250114 02:54:39.901993 960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550e84ac4d1b4ef58dfdea6e552ee940" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3"
I20250114 02:54:39.902484 960 raft_consensus.cc:3054] T 550e84ac4d1b4ef58dfdea6e552ee940 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:39.903076 465 catalog_manager.cc:5526] T a83bae91c29b463aa78412ce2e152ade P 819f9e3412e54d44abb11d8b3ca877c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 525198593c8e47afaee54e74267b87e3 (127.0.107.66). New cstate: current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } }
I20250114 02:54:39.915123 2447 consensus_queue.cc:1035] T e7ca0bd1dbd542de9ee11e6c72522ae0 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250114 02:54:39.916520 898 leader_election.cc:343] T 6d741bfd00784d2d830be3ebe50dc7c4 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:39.921911 960 raft_consensus.cc:2463] T 550e84ac4d1b4ef58dfdea6e552ee940 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d6c2c58bb54e0eafb365625ac33871 in term 1.
I20250114 02:54:39.921957 465 catalog_manager.cc:5526] T 68981913851243a7a1875fc99efb3c49 P 819f9e3412e54d44abb11d8b3ca877c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 525198593c8e47afaee54e74267b87e3 (127.0.107.66). New cstate: current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } }
I20250114 02:54:39.922853 1040 leader_election.cc:304] T 550e84ac4d1b4ef58dfdea6e552ee940 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525198593c8e47afaee54e74267b87e3, 90d6c2c58bb54e0eafb365625ac33871; no voters:
I20250114 02:54:39.923067 465 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 68981913851243a7a1875fc99efb3c49 (table default.Table_5e2861a7715a47039e64f1a717c57997 [id=683a61c5af09453e8027e9cb5b5b80a7]). Expected version 1 got 0
I20250114 02:54:39.923563 2510 raft_consensus.cc:2798] T 550e84ac4d1b4ef58dfdea6e552ee940 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:54:39.923717 465 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 719188f639724851a5fc2b37cb60fdbb (table default.Table_8f8a18d7868c44bbb4d66a0692b08186 [id=e6292728826f437cb75c524a8aef2e87]). Expected version 1 got 0
I20250114 02:54:39.924192 2510 raft_consensus.cc:695] T 550e84ac4d1b4ef58dfdea6e552ee940 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Becoming Leader. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Running, Role: LEADER
I20250114 02:54:39.924939 2510 consensus_queue.cc:237] T 550e84ac4d1b4ef58dfdea6e552ee940 P 90d6c2c58bb54e0eafb365625ac33871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:39.926033 465 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet fc406eab968745d893f1a8538ff61f37 (table default.Table_7367bdf975b243aab9651828ad630606 [id=e1ca4114e55d48839dfa2f76624a4234]). Expected version 1 got 0
I20250114 02:54:39.928475 465 catalog_manager.cc:5526] T 2e17f619ed2745078541cc1c924aff96 P 819f9e3412e54d44abb11d8b3ca877c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65). New cstate: current_term: 1 leader_uuid: "819f9e3412e54d44abb11d8b3ca877c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } }
I20250114 02:54:39.930966 1207 raft_consensus.cc:357] T 2d4eaf90be6d497d8a3c4d9ac418f4e9 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:39.931671 1207 raft_consensus.cc:383] T 2d4eaf90be6d497d8a3c4d9ac418f4e9 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:39.931924 1207 raft_consensus.cc:738] T 2d4eaf90be6d497d8a3c4d9ac418f4e9 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
W20250114 02:54:39.933650 898 leader_election.cc:343] T 6d741bfd00784d2d830be3ebe50dc7c4 P 525198593c8e47afaee54e74267b87e3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:54:39.937037 1205 tablet_bootstrap.cc:492] T a2c3f3a89b3d47f1b0dfacfac8218805 P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:39.937420 1205 ts_tablet_manager.cc:1397] T a2c3f3a89b3d47f1b0dfacfac8218805 P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.208s user 0.008s sys 0.006s
I20250114 02:54:39.939503 1205 raft_consensus.cc:357] T a2c3f3a89b3d47f1b0dfacfac8218805 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:39.945056 2625 ts_tablet_manager.cc:1918] T 3161f75bc4e84000af022faa9804c810 P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:54:39.945420 2625 log.cc:1198] T 3161f75bc4e84000af022faa9804c810 P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/3161f75bc4e84000af022faa9804c810
I20250114 02:54:39.946044 2625 ts_tablet_manager.cc:1939] T 3161f75bc4e84000af022faa9804c810 P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
I20250114 02:54:39.951719 1207 consensus_queue.cc:260] T 2d4eaf90be6d497d8a3c4d9ac418f4e9 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:39.953189 465 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet df045d80d8ce4ff194a6d2039a44bce6 (table default.Table_2befe573a994473abe0556ba7e713686 [id=da0258578bb34ae29551fc593bf7d0b5]). Expected version 1 got 0
I20250114 02:54:39.954401 465 catalog_manager.cc:5526] T a2d68eccaaa64511a9ae36f2b07a7b59 P 819f9e3412e54d44abb11d8b3ca877c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 525198593c8e47afaee54e74267b87e3 (127.0.107.66). New cstate: current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } }
I20250114 02:54:39.940076 1205 raft_consensus.cc:383] T a2c3f3a89b3d47f1b0dfacfac8218805 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:39.961087 1205 raft_consensus.cc:738] T a2c3f3a89b3d47f1b0dfacfac8218805 P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:39.956139 465 catalog_manager.cc:5526] T 0b8989d340fa4ae992ef97e226d41110 P 819f9e3412e54d44abb11d8b3ca877c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67). New cstate: current_term: 1 leader_uuid: "90d6c2c58bb54e0eafb365625ac33871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } }
I20250114 02:54:39.965437 465 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 797d6cdf36704c98aa106aac87a945ca (table default.Table_5a9480c6474848d6b400e7dd274d15b0 [id=a17ad30ef32f4cbcb22b5a7d1c7096ff]). Expected version 1 got 0
I20250114 02:54:39.966032 465 catalog_manager.cc:5526] T 5d545ac27a7f49faace0655713405cea P 819f9e3412e54d44abb11d8b3ca877c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 525198593c8e47afaee54e74267b87e3 (127.0.107.66). New cstate: current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } }
I20250114 02:54:39.966959 465 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet 5d545ac27a7f49faace0655713405cea (table default.Table_b4dc2961391c43fcb08fa062d4e45b3c [id=491b34b03ce14bd9be16e04915468928]). Expected version 1 got 0
I20250114 02:54:39.967664 465 catalog_manager.cc:5618] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397) does not have the latest schema for tablet d2a7b2bee5c1440fb705b172c18a9247 (table default.Table_3c0c8ba6eec34be4a6ea569d66038106 [id=4d2f7fabf8a44155a586a2b79a8fca53]). Expected version 1 got 0
I20250114 02:54:39.975841 2294 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [term 2 FOLLOWER]: Refusing update from remote peer 687d4ff6d3cb42cf90a965070b0d8a70: Log matching property violated. Preceding OpId in replica: term: 1 index: 406. Preceding OpId from leader: term: 2 index: 441. (index mismatch)
I20250114 02:54:39.977506 2525 raft_consensus.cc:491] T 46eb13912f754bf1b1fe9d0ad429c3c1 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:39.977433 2671 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87b3db09ac87436e9a60d6de5fedaff5" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 441, Last known committed idx: 405, Time since last communication: 0.000s
I20250114 02:54:39.978145 2525 raft_consensus.cc:513] T 46eb13912f754bf1b1fe9d0ad429c3c1 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:39.980587 2525 leader_election.cc:290] T 46eb13912f754bf1b1fe9d0ad429c3c1 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037), 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397)
I20250114 02:54:39.981406 960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46eb13912f754bf1b1fe9d0ad429c3c1" candidate_uuid: "90d6c2c58bb54e0eafb365625ac33871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525198593c8e47afaee54e74267b87e3" is_pre_election: true
I20250114 02:54:39.983479 960 raft_consensus.cc:2388] T 46eb13912f754bf1b1fe9d0ad429c3c1 P 525198593c8e47afaee54e74267b87e3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 90d6c2c58bb54e0eafb365625ac33871 in current term 1: Already voted for candidate 819f9e3412e54d44abb11d8b3ca877c2 in this term.
I20250114 02:54:39.972479 1205 consensus_queue.cc:260] T a2c3f3a89b3d47f1b0dfacfac8218805 P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:39.988725 1207 ts_tablet_manager.cc:1428] T 2d4eaf90be6d497d8a3c4d9ac418f4e9 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.090s user 0.004s sys 0.003s
I20250114 02:54:39.990111 1205 ts_tablet_manager.cc:1428] T a2c3f3a89b3d47f1b0dfacfac8218805 P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.052s user 0.003s sys 0.002s
I20250114 02:54:39.998391 1205 tablet_bootstrap.cc:492] T ab9a85db127b4d82ae1161717ed6006a P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:40.011380 2625 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 377) took 1153 ms. Trace:
I20250114 02:54:40.012455 2662 raft_consensus.cc:491] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:40.016731 2625 rpcz_store.cc:276] 0114 02:54:38.858188 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:38.858309 (+ 121us) service_pool.cc:225] Handling call
0114 02:54:39.655562 (+797253us) ts_tablet_manager.cc:1145] Deleting tablet 3161f75bc4e84000af022faa9804c810
0114 02:54:39.655601 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:39.812023 (+156422us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:39.944795 (+132772us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:40.011305 (+ 66510us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":69159,"tablet-delete.queue_time_us":796699}
I20250114 02:54:40.018831 1207 tablet_bootstrap.cc:492] T a2c3f3a89b3d47f1b0dfacfac8218805 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:40.025612 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet 3161f75bc4e84000af022faa9804c810 (table default.Table_35a78676616e4b20906b262c9d0c0d9e [id=e984149e9da444f1b2802e3f35a60fd6]) successfully deleted
I20250114 02:54:40.027760 2625 tablet_replica.cc:331] T ed7f575653c848d7b076c17c38c4a244 P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:40.052366 1205 tablet_bootstrap.cc:654] T ab9a85db127b4d82ae1161717ed6006a P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:40.054602 2625 raft_consensus.cc:2238] T ed7f575653c848d7b076c17c38c4a244 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Raft consensus shutting down.
I20250114 02:54:40.055684 2625 pending_rounds.cc:62] T ed7f575653c848d7b076c17c38c4a244 P 90d6c2c58bb54e0eafb365625ac33871: Trying to abort 1 pending ops.
I20250114 02:54:40.012877 2662 raft_consensus.cc:513] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b303b7df086340a0b5ad6373940ae750" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 45367 } } peers { permanent_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } } peers { permanent_uuid: "87b3db09ac87436e9a60d6de5fedaff5" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } }
I20250114 02:54:40.058686 2662 leader_election.cc:290] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 687d4ff6d3cb42cf90a965070b0d8a70 (127.0.107.125:46837), 87b3db09ac87436e9a60d6de5fedaff5 (127.0.107.124:39105)
I20250114 02:54:40.056133 2625 pending_rounds.cc:69] T ed7f575653c848d7b076c17c38c4a244 P 90d6c2c58bb54e0eafb365625ac33871: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114028131953569792 op_type: NO_OP noop_request { }
I20250114 02:54:40.060312 481 raft_consensus.cc:1270] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750 [term 2 FOLLOWER]: Refusing update from remote peer 687d4ff6d3cb42cf90a965070b0d8a70: Log matching property violated. Preceding OpId in replica: term: 1 index: 441. Preceding OpId from leader: term: 2 index: 441. (term mismatch)
I20250114 02:54:40.060655 481 pending_rounds.cc:77] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Aborting all ops after (but not including) 440
I20250114 02:54:40.060940 481 pending_rounds.cc:99] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: Aborting uncommitted WRITE_OP operation due to leader change: 1.441
W20250114 02:54:40.061806 468 catalog_manager.cc:6064] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: An error occurred while updating sys-tables: Aborted: Op aborted by new leader
I20250114 02:54:40.061805 556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b303b7df086340a0b5ad6373940ae750" candidate_term: 3 candidate_status { last_received { term: 1 index: 441 } } ignore_live_leader: false dest_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" is_pre_election: true
I20250114 02:54:40.063066 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b303b7df086340a0b5ad6373940ae750" candidate_term: 3 candidate_status { last_received { term: 1 index: 441 } } ignore_live_leader: false dest_uuid: "87b3db09ac87436e9a60d6de5fedaff5" is_pre_election: true
I20250114 02:54:40.065375 455 leader_election.cc:304] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b303b7df086340a0b5ad6373940ae750; no voters: 687d4ff6d3cb42cf90a965070b0d8a70, 87b3db09ac87436e9a60d6de5fedaff5
I20250114 02:54:40.066634 2662 raft_consensus.cc:2743] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 02:54:40.067902 2671 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b303b7df086340a0b5ad6373940ae750" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 45367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 441, Last known committed idx: 440, Time since last communication: 0.000s
I20250114 02:54:40.072438 2662 sys_catalog.cc:455] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 687d4ff6d3cb42cf90a965070b0d8a70. Latest consensus state: current_term: 2 leader_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b303b7df086340a0b5ad6373940ae750" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 45367 } } peers { permanent_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } } peers { permanent_uuid: "87b3db09ac87436e9a60d6de5fedaff5" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } } }
I20250114 02:54:40.073351 2662 sys_catalog.cc:458] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750 [sys.catalog]: This master's current role is: FOLLOWER
I20250114 02:54:40.080152 2625 raft_consensus.cc:2883] T ed7f575653c848d7b076c17c38c4a244 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250114 02:54:40.082686 2625 raft_consensus.cc:2267] T ed7f575653c848d7b076c17c38c4a244 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:40.088583 2625 ts_tablet_manager.cc:1905] T ed7f575653c848d7b076c17c38c4a244 P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:40.102974 683 catalog_manager.cc:1603] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
E20250114 02:54:40.005259 465 catalog_manager.cc:5681] Error updating tablets from 819f9e3412e54d44abb11d8b3ca877c2: Illegal state: replica b303b7df086340a0b5ad6373940ae750 is not leader of this config: current role FOLLOWER. Tablet report was: is_incremental: true updated_tablets { tablet_id: "550e84ac4d1b4ef58dfdea6e552ee940" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "038ea1ea231e490ba27ecd5430883058" state: RUNNING consensus_state { current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "e7ca0bd1dbd542de9ee11e6c72522ae0" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "e06daadd346d45969ad5efcf03bc0870" state: RUNNING consensus_state { current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "ed109b49f20642f988161a3df7872419" state: RUNNING consensus_state { current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "6d741bfd00784d2d830be3ebe50dc7c4" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "a2d68eccaaa64511a9ae36f2b07a7b59" state: RUNNING consensus_state { current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "0b8361cd538046e0bcb4aac821748f67" state: RUNNING consensus_state { current_term: 1 leader_uuid: "90d6c2c58bb54e0eafb365625ac33871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "612987f3e4ea4a02932ceefa0d740425" state: RUNNING consensus_state { current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "797d6cdf36704c98aa106aac87a945ca" state: RUNNING consensus_state { current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "5d545ac27a7f49faace0655713405cea" state: RUNNING consensus_state { current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "0b8989d340fa4ae992ef97e226d41110" state: RUNNING consensus_state { current_term: 1 leader_uuid: "90d6c2c58bb54e0eafb365625ac33871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "f40c66314342453b898da3ac0e7e86bf" state: RUNNING consensus_state { current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "d56255ca806d4e2cac09fbb5e1b1efbd" state: RUNNING consensus_state { current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "df045d80d8ce4ff194a6d2039a44bce6" state: RUNNING consensus_state { current_term: 1 leader_uuid: "819f9e3412e54d44abb11d8b3ca877c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 26841 live_row_count: 0 } } updated_tablets { tablet_id: "2e17f619ed2745078541cc1c924aff96" state: RUNNING consensus_state { current_term: 1 leader_uuid: "819f9e3412e54d44abb11d8b3ca877c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { } } updated_tablets { tablet_id: "9bfd8fd6d2104fb4a6484cb8c98981bd" state: RUNNING consensus_state { current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "fc406eab968745d893f1a8538ff61f37" state: RUNNING consensus_state { current_term: 2 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "f464eabfbc0945fa8a5583f00c1dab40" state: RUNNING consensus_state { current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "6e2fb05f72b743e2a02ed36b63f80b23" state: RUNNING consensus_state { current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "719188f639724851a5fc2b37cb60fdbb" state: RUNNING consensus_state { current_term: 2 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "46eb13912f754bf1b1fe9d0ad429c3c1" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "4bb9b429af92446bbe13894d5dfe4c47" state: RUNNING consensus_state { current_term: 1 leader_uuid: "819f9e3412e54d44abb11d8b3ca877c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } } } schema_version: 2 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27285 live_row_count: 0 } } updated_tablets { tablet_id: "97de018391f84b0ebd3130861681b7b5" state: RUNNING consensus_state { current_term: 1 leader_uuid: "819f9e3412e54d44abb11d8b3ca877c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: UNKNOWN } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { } } updated_tablets { tablet_id: "a83bae91c29b463aa78412ce2e152ade" state: RUNNING consensus_state { current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "a52d6749ff834b62a8e20a2f45935240" state: RUNNING consensus_state { current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "15704785f56e43a3af69f181c75877ce" state: RUNNING consensus_state { current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "044832851f804208b0a371eb0857dbf3" state: RUNNING consensus_state { current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "ddfbab181b1f474192c664cd0316037f" state: RUNNING consensus_state { current_term: 1 leader_uuid: "90d6c2c58bb54e0eafb365625ac33871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "92cd8eccacf34ab7a66b3bbc69df3697" state: RUNNING consensus_state { current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } } } schema_version: 2 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "42cee9e8e26147b593ed73fca6ac1183" state: RUNNING consensus_state { current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "68981913851243a7a1875fc99efb3c49" state: RUNNING consensus_state { current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "71bd00f7d8ac41839f36fd73997631ad" state: RUNNING consensus_state { current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "a2c3f3a89b3d47f1b0dfacfac8218805" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "0304df55431e42e1a999ed17dc35e6c0" state: RUNNING consensus_state { current_term: 1 leader_uuid: "90d6c2c58bb54e0eafb365625ac33871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "ce7d6088fc884a248918d2348adcaccf" state: RUNNING consensus_state { current_term: 1 leader_uuid: "90d6c2c58bb54e0eafb365625ac33871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "f2b6812e8c1d44529d0284be490df648" state: RUNNING consensus_state { current_term: 1 leader_uuid: "819f9e3412e54d44abb11d8b3ca877c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 26841 live_row_count: 0 } } updated_tablets { tablet_id: "6448eea21ab2422c97da2494658ded3f" state: RUNNING consensus_state { current_term: 1 leader_uuid: "90d6c2c58bb54e0eafb365625ac33871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } } } schema_version: 2 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "aad96f2507974023aaa988e6950feb7a" state: RUNNING consensus_state { current_term: 1 leader_uuid: "819f9e3412e54d44abb11d8b3ca877c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } } } schema_version: 2 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27063 live_row_count: 0 } } updated_tablets { tablet_id: "fcb58a05f980460884ed81a30a5d59c2" state: RUNNING consensus_state { current_term: 1 leader_uuid: "90d6c2c58bb54e0eafb365625ac33871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "d2a7b2bee5c1440fb705b172c18a9247" state: RUNNING consensus_state { current_term: 1 leader_uuid: "90d6c2c58bb54e0eafb365625ac33871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "967f4d5b944543d49e87645d069b5b55" state: RUNNING consensus_state { current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "a3fdab330d574c0283ac58bc9e450b87" state: RUNNING consensus_state { current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "38fe8829d98547fb907ef78099df3d7f" state: RUNNING consensus_state { current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config {
I20250114 02:54:40.117936 1205 tablet_bootstrap.cc:492] T ab9a85db127b4d82ae1161717ed6006a P 525198593c8e47afaee54e74267b87e3: No bootstrap required, opened a new log
I20250114 02:54:40.118328 1205 ts_tablet_manager.cc:1397] T ab9a85db127b4d82ae1161717ed6006a P 525198593c8e47afaee54e74267b87e3: Time spent bootstrapping tablet: real 0.120s user 0.004s sys 0.004s
I20250114 02:54:40.120445 1205 raft_consensus.cc:357] T ab9a85db127b4d82ae1161717ed6006a P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:40.111131 2662 sys_catalog.cc:455] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b303b7df086340a0b5ad6373940ae750" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 45367 } } peers { permanent_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } } peers { permanent_uuid: "87b3db09ac87436e9a60d6de5fedaff5" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } } }
I20250114 02:54:40.121042 1205 raft_consensus.cc:383] T ab9a85db127b4d82ae1161717ed6006a P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:40.121266 1205 raft_consensus.cc:738] T ab9a85db127b4d82ae1161717ed6006a P 525198593c8e47afaee54e74267b87e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525198593c8e47afaee54e74267b87e3, State: Initialized, Role: FOLLOWER
I20250114 02:54:40.121378 2662 sys_catalog.cc:458] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750 [sys.catalog]: This master's current role is: FOLLOWER
I20250114 02:54:40.129539 1207 tablet_bootstrap.cc:654] T a2c3f3a89b3d47f1b0dfacfac8218805 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:40.134501 2642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b303b7df086340a0b5ad6373940ae750" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 45367 } } peers { permanent_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } } peers { permanent_uuid: "87b3db09ac87436e9a60d6de5fedaff5" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } } }
I20250114 02:54:40.135311 2642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [sys.catalog]: This master's current role is: LEADER
W20250114 02:54:40.146482 465 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.65:49805 (request call id 63) took 1615 ms (client timeout 1000 ms). Trace:
I20250114 02:54:40.158082 1205 consensus_queue.cc:260] T ab9a85db127b4d82ae1161717ed6006a P 525198593c8e47afaee54e74267b87e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
W20250114 02:54:40.160729 465 rpcz_store.cc:269] 0114 02:54:38.530704 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:38.813802 (+283098us) service_pool.cc:225] Handling call
0114 02:54:40.005034 (+1191232us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:40.005053 (+ 19us) write_op.cc:454] Released schema lock
0114 02:54:40.146413 (+141360us) inbound_call.cc:173] Queueing failure response
Related trace 'op':
0114 02:54:40.004822 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica b303b7df086340a0b5ad6373940ae750 is not leader of this config: current role FOLLOWER)
0114 02:54:40.004860 (+ 38us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:40.004879 (+ 19us) write_op.cc:454] Released schema lock
0114 02:54:40.004898 (+ 19us) write_op.cc:335] FINISH: Op aborted
Metrics: {"reported_tablets":53,"child_traces":[["op",{}]]}
I20250114 02:54:40.173758 1205 ts_tablet_manager.cc:1428] T ab9a85db127b4d82ae1161717ed6006a P 525198593c8e47afaee54e74267b87e3: Time spent starting tablet: real 0.055s user 0.005s sys 0.001s
I20250114 02:54:40.174618 1205 tablet_bootstrap.cc:492] T ddad7584337345109ce102702008ab37 P 525198593c8e47afaee54e74267b87e3: Bootstrap starting.
I20250114 02:54:40.176880 2625 ts_tablet_manager.cc:1918] T ed7f575653c848d7b076c17c38c4a244 P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:40.177369 2625 log.cc:1198] T ed7f575653c848d7b076c17c38c4a244 P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/ed7f575653c848d7b076c17c38c4a244
I20250114 02:54:40.178187 2625 ts_tablet_manager.cc:1939] T ed7f575653c848d7b076c17c38c4a244 P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
I20250114 02:54:40.180446 2625 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 378) took 1322 ms. Trace:
I20250114 02:54:40.180896 2625 rpcz_store.cc:276] 0114 02:54:38.858461 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:38.871548 (+ 13087us) service_pool.cc:225] Handling call
0114 02:54:40.025678 (+1154130us) ts_tablet_manager.cc:1145] Deleting tablet ed7f575653c848d7b076c17c38c4a244
0114 02:54:40.027557 (+ 1879us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:40.151121 (+123564us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:40.176657 (+ 25536us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:40.180389 (+ 3732us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25823,"tablet-delete.queue_time_us":1153547}
I20250114 02:54:40.182300 2625 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 379) took 1309 ms. Trace:
I20250114 02:54:40.182540 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet ed7f575653c848d7b076c17c38c4a244 (table default.Table_f842a61f536940588cd46bedc206f2b3 [id=954669573a3c4a97a488805e05b0b220]) successfully deleted
I20250114 02:54:40.182646 2625 rpcz_store.cc:276] 0114 02:54:38.872451 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:38.872545 (+ 94us) service_pool.cc:225] Handling call
0114 02:54:40.182043 (+1309498us) ts_tablet_manager.cc:1145] Deleting tablet c86562688d5b4fffb71cbdf7be0afc37
0114 02:54:40.182085 (+ 42us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:40.182252 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1308997}
W20250114 02:54:40.183986 458 catalog_manager.cc:4839] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet c86562688d5b4fffb71cbdf7be0afc37 because the tablet was not found. No further retry: Not found: Tablet not found: c86562688d5b4fffb71cbdf7be0afc37
I20250114 02:54:40.191529 466 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 141cede72b8741a8b0615d7c6bb823bb (table default.Table_4364d1df16d149d2ba10469ce7288d4e [id=13fc996d911b430290e37a94dd44b1ab]). Expected version 1 got 0
I20250114 02:54:40.195756 1207 tablet_bootstrap.cc:492] T a2c3f3a89b3d47f1b0dfacfac8218805 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:40.196148 1207 ts_tablet_manager.cc:1397] T a2c3f3a89b3d47f1b0dfacfac8218805 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.177s user 0.011s sys 0.000s
I20250114 02:54:40.198297 1207 raft_consensus.cc:357] T a2c3f3a89b3d47f1b0dfacfac8218805 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:40.200351 1207 raft_consensus.cc:383] T a2c3f3a89b3d47f1b0dfacfac8218805 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:40.200605 1207 raft_consensus.cc:738] T a2c3f3a89b3d47f1b0dfacfac8218805 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:40.201056 2625 tablet_replica.cc:331] T 97f4e061d3b94ee9982410b7ce135453 P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:40.201200 1207 consensus_queue.cc:260] T a2c3f3a89b3d47f1b0dfacfac8218805 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:40.205617 1207 ts_tablet_manager.cc:1428] T a2c3f3a89b3d47f1b0dfacfac8218805 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.009s user 0.002s sys 0.003s
W20250114 02:54:40.206324 468 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 797d6cdf36704c98aa106aac87a945ca (table default.Table_5a9480c6474848d6b400e7dd274d15b0 [id=a17ad30ef32f4cbcb22b5a7d1c7096ff], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 797d6cdf36704c98aa106aac87a945ca consensus state has no leader
W20250114 02:54:40.208827 468 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.65:49805 (request call id 61) took 4003 ms (client timeout 1000 ms). Trace:
I20250114 02:54:40.212103 1099 raft_consensus.cc:3054] T 2d4eaf90be6d497d8a3c4d9ac418f4e9 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:54:40.218930 2675 consensus_queue.cc:1035] T 2d4eaf90be6d497d8a3c4d9ac418f4e9 P 525198593c8e47afaee54e74267b87e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:54:40.201216 466 catalog_manager.cc:5526] T a83bae91c29b463aa78412ce2e152ade P 525198593c8e47afaee54e74267b87e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 525198593c8e47afaee54e74267b87e3 (127.0.107.66). New cstate: current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } }
I20250114 02:54:40.244695 466 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet a52d6749ff834b62a8e20a2f45935240 (table default.Table_1c9b7896941e4289b1c12e20c210c2c1 [id=b3ee15b9f10f4d489edbde3af9034d25]). Expected version 1 got 0
I20250114 02:54:40.245292 2625 raft_consensus.cc:2238] T 97f4e061d3b94ee9982410b7ce135453 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:40.245843 2625 raft_consensus.cc:2267] T 97f4e061d3b94ee9982410b7ce135453 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:40.245441 466 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet df045d80d8ce4ff194a6d2039a44bce6 (table default.Table_2befe573a994473abe0556ba7e713686 [id=da0258578bb34ae29551fc593bf7d0b5]). Expected version 1 got 0
I20250114 02:54:40.246641 466 catalog_manager.cc:5526] T 612987f3e4ea4a02932ceefa0d740425 P 525198593c8e47afaee54e74267b87e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 525198593c8e47afaee54e74267b87e3 (127.0.107.66). New cstate: current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:40.248250 2625 ts_tablet_manager.cc:1905] T 97f4e061d3b94ee9982410b7ce135453 P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:40.232488 2687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 687d4ff6d3cb42cf90a965070b0d8a70. Latest consensus state: current_term: 2 leader_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b303b7df086340a0b5ad6373940ae750" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 45367 } } peers { permanent_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } } peers { permanent_uuid: "87b3db09ac87436e9a60d6de5fedaff5" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } } }
I20250114 02:54:40.250138 2687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [sys.catalog]: This master's current role is: FOLLOWER
I20250114 02:54:40.249308 466 catalog_manager.cc:5526] T 68981913851243a7a1875fc99efb3c49 P 525198593c8e47afaee54e74267b87e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 525198593c8e47afaee54e74267b87e3 (127.0.107.66). New cstate: current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: UNKNOWN } } }
W20250114 02:54:40.209311 468 rpcz_store.cc:269] 0114 02:54:36.205901 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:36.428817 (+222916us) service_pool.cc:225] Handling call
0114 02:54:40.208757 (+3779940us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 02:54:39.039987 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0114 02:54:39.040455 (+ 468us) write_op.cc:432] Acquiring schema lock in shared mode
0114 02:54:39.040481 (+ 26us) write_op.cc:435] Acquired schema lock
0114 02:54:39.040496 (+ 15us) tablet.cc:659] Decoding operations
0114 02:54:39.040826 (+ 330us) write_op.cc:620] Acquiring the partition lock for write op
0114 02:54:39.040870 (+ 44us) write_op.cc:641] Partition lock acquired for write op
0114 02:54:39.040894 (+ 24us) tablet.cc:682] Acquiring locks for 1 operations
0114 02:54:39.041044 (+ 150us) tablet.cc:698] Row locks acquired
0114 02:54:39.041070 (+ 26us) write_op.cc:260] PREPARE: finished
0114 02:54:39.041176 (+ 106us) write_op.cc:270] Start()
0114 02:54:39.041255 (+ 79us) write_op.cc:276] Timestamp: P: 1736823279041097 usec, L: 0
0114 02:54:39.041272 (+ 17us) op_driver.cc:348] REPLICATION: starting
0114 02:54:39.041812 (+ 540us) log.cc:844] Serialized 356 byte log entry
0114 02:54:40.061354 (+1019542us) op_driver.cc:464] REPLICATION: finished
0114 02:54:40.061412 (+ 58us) op_driver.cc:389] HandleFailure(Aborted: Op aborted by new leader)
0114 02:54:40.061446 (+ 34us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:40.061581 (+ 135us) write_op.cc:454] Released schema lock
0114 02:54:40.061614 (+ 33us) write_op.cc:335] FINISH: Op aborted
Metrics: {"mutex_wait_us":100537,"reported_tablets":45,"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":122,"prepare.run_cpu_time_us":2037,"prepare.run_wall_time_us":3218,"replication_time_us":1019923}]]}
I20250114 02:54:40.262671 1207 tablet_bootstrap.cc:492] T 6d741bfd00784d2d830be3ebe50dc7c4 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:40.261806 466 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 68981913851243a7a1875fc99efb3c49 (table default.Table_5e2861a7715a47039e64f1a717c57997 [id=683a61c5af09453e8027e9cb5b5b80a7]). Expected version 1 got 0
I20250114 02:54:40.263629 466 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 719188f639724851a5fc2b37cb60fdbb (table default.Table_8f8a18d7868c44bbb4d66a0692b08186 [id=e6292728826f437cb75c524a8aef2e87]). Expected version 1 got 0
I20250114 02:54:40.264700 466 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet fc406eab968745d893f1a8538ff61f37 (table default.Table_7367bdf975b243aab9651828ad630606 [id=e1ca4114e55d48839dfa2f76624a4234]). Expected version 1 got 0
I20250114 02:54:40.272781 466 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 30a56fb2d941470590e17f37b90aef26 (table default.Table_0e0710e3f43e4e859ad32662704ee41c [id=66082998119e4c43a666fb43087fc622]). Expected version 1 got 0
I20250114 02:54:40.273969 466 catalog_manager.cc:5526] T a2d68eccaaa64511a9ae36f2b07a7b59 P 525198593c8e47afaee54e74267b87e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 525198593c8e47afaee54e74267b87e3 (127.0.107.66). New cstate: current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:40.281347 2673 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250114 02:54:40.277235 466 catalog_manager.cc:5526] T 0b8989d340fa4ae992ef97e226d41110 P 525198593c8e47afaee54e74267b87e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67). New cstate: current_term: 1 leader_uuid: "90d6c2c58bb54e0eafb365625ac33871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } }
I20250114 02:54:40.285804 466 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 797d6cdf36704c98aa106aac87a945ca (table default.Table_5a9480c6474848d6b400e7dd274d15b0 [id=a17ad30ef32f4cbcb22b5a7d1c7096ff]). Expected version 1 got 0
I20250114 02:54:40.286433 466 catalog_manager.cc:5526] T 5d545ac27a7f49faace0655713405cea P 525198593c8e47afaee54e74267b87e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 525198593c8e47afaee54e74267b87e3 (127.0.107.66). New cstate: current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } }
I20250114 02:54:40.287776 466 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 5d545ac27a7f49faace0655713405cea (table default.Table_b4dc2961391c43fcb08fa062d4e45b3c [id=491b34b03ce14bd9be16e04915468928]). Expected version 1 got 0
I20250114 02:54:40.288410 466 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet d2a7b2bee5c1440fb705b172c18a9247 (table default.Table_3c0c8ba6eec34be4a6ea569d66038106 [id=4d2f7fabf8a44155a586a2b79a8fca53]). Expected version 1 got 0
I20250114 02:54:40.327492 2625 ts_tablet_manager.cc:1918] T 97f4e061d3b94ee9982410b7ce135453 P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:54:40.347261 2671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b303b7df086340a0b5ad6373940ae750" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 45367 } } peers { permanent_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } } peers { permanent_uuid: "87b3db09ac87436e9a60d6de5fedaff5" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } } }
I20250114 02:54:40.351527 2671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 687d4ff6d3cb42cf90a965070b0d8a70 [sys.catalog]: This master's current role is: LEADER
I20250114 02:54:40.336676 2687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b303b7df086340a0b5ad6373940ae750" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 45367 } } peers { permanent_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } } peers { permanent_uuid: "87b3db09ac87436e9a60d6de5fedaff5" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } } }
I20250114 02:54:40.356542 2687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [sys.catalog]: This master's current role is: FOLLOWER
I20250114 02:54:40.363893 2625 log.cc:1198] T 97f4e061d3b94ee9982410b7ce135453 P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/97f4e061d3b94ee9982410b7ce135453
I20250114 02:54:40.352010 1207 tablet_bootstrap.cc:654] T 6d741bfd00784d2d830be3ebe50dc7c4 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:40.366269 2625 ts_tablet_manager.cc:1939] T 97f4e061d3b94ee9982410b7ce135453 P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
I20250114 02:54:40.369040 1205 tablet_bootstrap.cc:654] T ddad7584337345109ce102702008ab37 P 525198593c8e47afaee54e74267b87e3: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:40.383095 1207 tablet_bootstrap.cc:492] T 6d741bfd00784d2d830be3ebe50dc7c4 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:40.384541 1207 ts_tablet_manager.cc:1397] T 6d741bfd00784d2d830be3ebe50dc7c4 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.122s user 0.007s sys 0.003s
I20250114 02:54:40.387290 1207 raft_consensus.cc:357] T 6d741bfd00784d2d830be3ebe50dc7c4 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:40.400334 1207 raft_consensus.cc:383] T 6d741bfd00784d2d830be3ebe50dc7c4 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:40.400699 1207 raft_consensus.cc:738] T 6d741bfd00784d2d830be3ebe50dc7c4 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:40.401432 1207 consensus_queue.cc:260] T 6d741bfd00784d2d830be3ebe50dc7c4 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } }
I20250114 02:54:40.410403 1207 ts_tablet_manager.cc:1428] T 6d741bfd00784d2d830be3ebe50dc7c4 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.025s user 0.004s sys 0.001s
W20250114 02:54:40.407625 1042 connection.cc:537] client connection to 127.0.107.65:39397 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250114 02:54:40.412536 1042 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250114 02:54:40.412587 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_ceb42359708442c294b3f50aa6eadcc6 [id=0007a4e82dd44243a0db8a29a16350b0]
I20250114 02:54:40.415613 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_c62ee80bacab484cb74431dcb4723918 [id=01f1fa2a006c4bd4bc22e6648a2152dc]
I20250114 02:54:40.421948 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_d1de82caa5b94791b6c7dfed1c376e8a [id=03c775a54e9e443cbc4fab853bd6d49c]
I20250114 02:54:40.423449 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_26355cc977e54afca705ed400dae8207 [id=0436dc9f10d74e7eb434ac41f704af39]
I20250114 02:54:40.435827 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_c305caaf6328407e9e7e0100432f0e9d [id=0c56c93c919743228d2e35b39fcc805f]
I20250114 02:54:40.437561 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_4c3843233dda407395ab9652d5870bc6 [id=0d173fdb502f482fb86b6d3e5fdc66f1]
W20250114 02:54:40.437868 455 connection.cc:537] server connection from 127.0.107.65:49805 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250114 02:54:40.438015 898 connection.cc:537] client connection to 127.0.107.65:39397 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250114 02:54:40.438410 898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250114 02:54:40.439216 458 connection.cc:537] client connection to 127.0.107.65:39397 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250114 02:54:40.439558 458 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 4 similar messages]
I20250114 02:54:40.439990 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_5623f4fcb2fb42ef8cf628812939a998 [id=0f4fef7ce07a494a811442a3762b0443]
I20250114 02:54:40.441362 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_82cba210b70a491290d9ec065423f7c1 [id=12254cb2191f486cb23b978958c999f9]
I20250114 02:54:40.442782 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_ee814dd266e64286864db0eac9e11664 [id=13151866bf2b4f44b1dd5c92380a4d43]
I20250114 02:54:40.444172 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_4364d1df16d149d2ba10469ce7288d4e [id=13fc996d911b430290e37a94dd44b1ab]
I20250114 02:54:40.444651 429 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrCUXNq/build/tsan/bin/kudu with pid 873
W20250114 02:54:40.451153 2704 negotiation.cc:337] Failed RPC negotiation. Trace:
0114 02:54:40.431764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.107.65:39397 (local address 127.0.107.67:37129)
0114 02:54:40.450188 (+ 18424us) negotiation.cc:107] Waiting for socket to connect
0114 02:54:40.450452 (+ 264us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.107.65:39397: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":17546,"threads_started":1}
I20250114 02:54:40.451496 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_6d9d09d06ff34d9197882e2f88179301 [id=17a03e2b5b374954963af7fe597dbced]
W20250114 02:54:40.454246 1042 consensus_peers.cc:487] T 45301f8823ed4ec494e792d41f14f71f P 90d6c2c58bb54e0eafb365625ac33871 -> Peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Couldn't send request to peer 819f9e3412e54d44abb11d8b3ca877c2. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.65:39397: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20250114 02:54:40.454676 1042 consensus_peers.cc:487] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 90d6c2c58bb54e0eafb365625ac33871 -> Peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Couldn't send request to peer 819f9e3412e54d44abb11d8b3ca877c2. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.65:39397: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20250114 02:54:40.455021 1042 consensus_peers.cc:487] T e06daadd346d45969ad5efcf03bc0870 P 90d6c2c58bb54e0eafb365625ac33871 -> Peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Couldn't send request to peer 819f9e3412e54d44abb11d8b3ca877c2. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.65:39397: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20250114 02:54:40.455343 1042 leader_election.cc:336] T 550e84ac4d1b4ef58dfdea6e552ee940 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Network error: Client connection negotiation failed: client connection to 127.0.107.65:39397: connect: Connection reset by peer (error 104)
W20250114 02:54:40.455904 1042 leader_election.cc:336] T 550e84ac4d1b4ef58dfdea6e552ee940 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Network error: Client connection negotiation failed: client connection to 127.0.107.65:39397: connect: Connection reset by peer (error 104)
I20250114 02:54:40.456080 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_4150850ec6774c4a82d174648a35d1b4 [id=180820378e5f4a0899e6e24aa949b8fb]
I20250114 02:54:40.460362 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_9da872c9a22e4d07947df8db0524e61f [id=1f964e9ed159449580fe5326673c18ab]
W20250114 02:54:40.461606 1042 consensus_peers.cc:487] T 0304df55431e42e1a999ed17dc35e6c0 P 90d6c2c58bb54e0eafb365625ac33871 -> Peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Couldn't send request to peer 819f9e3412e54d44abb11d8b3ca877c2. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.65:39397: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20250114 02:54:40.461966 1042 leader_election.cc:336] T 46eb13912f754bf1b1fe9d0ad429c3c1 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Network error: Client connection negotiation failed: client connection to 127.0.107.65:39397: connect: Connection reset by peer (error 104)
I20250114 02:54:40.462308 1042 leader_election.cc:304] T 46eb13912f754bf1b1fe9d0ad429c3c1 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90d6c2c58bb54e0eafb365625ac33871; no voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2
I20250114 02:54:40.462980 2406 raft_consensus.cc:3054] T 46eb13912f754bf1b1fe9d0ad429c3c1 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Advancing to term 1
W20250114 02:54:40.463137 1042 consensus_peers.cc:487] T 0b8989d340fa4ae992ef97e226d41110 P 90d6c2c58bb54e0eafb365625ac33871 -> Peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Couldn't send request to peer 819f9e3412e54d44abb11d8b3ca877c2. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.65:39397: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20250114 02:54:40.463554 1042 consensus_peers.cc:487] T ce7d6088fc884a248918d2348adcaccf P 90d6c2c58bb54e0eafb365625ac33871 -> Peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Couldn't send request to peer 819f9e3412e54d44abb11d8b3ca877c2. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.65:39397: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20250114 02:54:40.463904 1042 consensus_peers.cc:487] T 0b8361cd538046e0bcb4aac821748f67 P 90d6c2c58bb54e0eafb365625ac33871 -> Peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Couldn't send request to peer 819f9e3412e54d44abb11d8b3ca877c2. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.65:39397: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
I20250114 02:54:40.466877 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_c778a0b532b64bcdbffa5cca153bcfcc [id=225e06c5be3042bab04da940d4caa996]
I20250114 02:54:40.468909 2406 raft_consensus.cc:2743] T 46eb13912f754bf1b1fe9d0ad429c3c1 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:54:40.470399 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_6071c0da3a0d4ecc9e063e0bab9712de [id=226c86aee94b4aa3b1569e2d4784e842]
I20250114 02:54:40.474493 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_e4cc8164591444a4bf1e683f3a609d36 [id=22b7edcf4b7e4c31b3c23dc44ccf028e]
W20250114 02:54:40.475131 1042 consensus_peers.cc:487] T ddfbab181b1f474192c664cd0316037f P 90d6c2c58bb54e0eafb365625ac33871 -> Peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Couldn't send request to peer 819f9e3412e54d44abb11d8b3ca877c2. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.65:39397: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20250114 02:54:40.475591 1042 consensus_peers.cc:487] T fcb58a05f980460884ed81a30a5d59c2 P 90d6c2c58bb54e0eafb365625ac33871 -> Peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Couldn't send request to peer 819f9e3412e54d44abb11d8b3ca877c2. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.65:39397: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20250114 02:54:40.475955 1042 consensus_peers.cc:487] T be7aad41ccdd4ef9bcfe8a985155708f P 90d6c2c58bb54e0eafb365625ac33871 -> Peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Couldn't send request to peer 819f9e3412e54d44abb11d8b3ca877c2. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.65:39397: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
I20250114 02:54:40.476820 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_945582460ea34007972cb70eb35e608a [id=22dcb9796ed94aba85494dfd591d9542]
I20250114 02:54:40.477691 2625 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 380) took 1604 ms. Trace:
I20250114 02:54:40.478029 2625 rpcz_store.cc:276] 0114 02:54:38.873312 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:38.873397 (+ 85us) service_pool.cc:225] Handling call
0114 02:54:40.200907 (+1327510us) ts_tablet_manager.cc:1145] Deleting tablet 97f4e061d3b94ee9982410b7ce135453
0114 02:54:40.200949 (+ 42us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:40.271919 (+ 70970us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:40.327269 (+ 55350us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:40.477627 (+150358us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1327044}
I20250114 02:54:40.480093 1207 tablet_bootstrap.cc:492] T 992bb16197b74be5b1dc884f29d9c240 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:40.481113 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_7176743e4429467481ed67254643b9f9 [id=22e634dc594d4962b404424977f2cd74]
I20250114 02:54:40.483249 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_b8047d6efb6845768a8d5192b794bd2a [id=24be7551945c40bc84b668b8747cf0e1]
I20250114 02:54:40.489184 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_017604272128419e9c9d1bba85e595cb [id=253dc2a0f94149a78b0f4ab001fba488]
I20250114 02:54:40.490561 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_990b8d5f1bd146d6b358457f954780c9 [id=26627ccdc808451fb9593ad7a6df6991]
W20250114 02:54:40.490644 1042 consensus_peers.cc:487] T ed109b49f20642f988161a3df7872419 P 90d6c2c58bb54e0eafb365625ac33871 -> Peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Couldn't send request to peer 819f9e3412e54d44abb11d8b3ca877c2. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.65:39397: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250114 02:54:40.479175 2625 tablet_replica.cc:331] T 99b940d872ce4781907517c40fe08781 P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:40.493172 2625 raft_consensus.cc:2238] T 99b940d872ce4781907517c40fe08781 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:54:40.493723 2625 raft_consensus.cc:2267] T 99b940d872ce4781907517c40fe08781 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:40.494501 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_0139fab1a81440309d9d7729c3e76c8a [id=2688bb1e2f4f4765986f738256d17f78]
I20250114 02:54:40.495604 2625 ts_tablet_manager.cc:1905] T 99b940d872ce4781907517c40fe08781 P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:40.497030 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_78eb79361f2c42538d83e9edcf6b51dc [id=2986220e36f94711ac450982e332f0f7]
I20250114 02:54:40.497495 1207 tablet_bootstrap.cc:654] T 992bb16197b74be5b1dc884f29d9c240 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:40.502172 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_579071b88792449ab70a258304f3356c [id=2f767d699397451f8d252f45a07a71d2]
W20250114 02:54:40.504520 1042 consensus_peers.cc:487] T d434cbc9daf549a6bdc04c8c76ead8a6 P 90d6c2c58bb54e0eafb365625ac33871 -> Peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Couldn't send request to peer 819f9e3412e54d44abb11d8b3ca877c2. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.65:39397: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250114 02:54:40.505611 458 catalog_manager.cc:4615] TS 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 99b940d872ce4781907517c40fe08781: Network error: Client connection negotiation failed: client connection to 127.0.107.65:39397: connect: Connection refused (error 111)
I20250114 02:54:40.506503 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet 97f4e061d3b94ee9982410b7ce135453 (table default.Table_6a4fe39298514187b65def5841d3a992 [id=5c5a7974ad2f4e58ba5c47a6eeaaac62]) successfully deleted
I20250114 02:54:40.512104 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_57f2197ca94b407c9fac253e8abc311d [id=312b160396c64566981a3b6325a8778d]
I20250114 02:54:40.513809 1207 tablet_bootstrap.cc:492] T 992bb16197b74be5b1dc884f29d9c240 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:40.513787 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_bf613453528d4aad91e5ecf042f8aa21 [id=347f25d3fd324991a401377d1327c93f]
I20250114 02:54:40.514298 1207 ts_tablet_manager.cc:1397] T 992bb16197b74be5b1dc884f29d9c240 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.034s user 0.009s sys 0.000s
I20250114 02:54:40.515309 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_428bd60f16824ac2b8bfaad6819f76e1 [id=35c5879da4b344118a377335fc6f83de]
I20250114 02:54:40.516716 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_539d51840e36495ab6185647ab9044e3 [id=35e5cff38891455aa271ec4f1939333e]
I20250114 02:54:40.516495 1207 raft_consensus.cc:357] T 992bb16197b74be5b1dc884f29d9c240 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:40.519851 1207 raft_consensus.cc:383] T 992bb16197b74be5b1dc884f29d9c240 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:40.527947 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_db23db370ae74af5b334c7e6b5e130c2 [id=38a5e63feaa1454abde9ee3e71414415]
I20250114 02:54:40.529045 1207 raft_consensus.cc:738] T 992bb16197b74be5b1dc884f29d9c240 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:40.530221 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_17190cd416734d35abcc8c6b354e5bb1 [id=3b8c1a8e39d84aef81cf505719161711]
I20250114 02:54:40.529913 1207 consensus_queue.cc:260] T 992bb16197b74be5b1dc884f29d9c240 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
W20250114 02:54:40.547106 1042 consensus_peers.cc:487] T 55b68586ef8b45439a7dd8dcd2efcd79 P 90d6c2c58bb54e0eafb365625ac33871 -> Peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Couldn't send request to peer 819f9e3412e54d44abb11d8b3ca877c2. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.65:39397: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
E20250114 02:54:40.387542 466 catalog_manager.cc:5681] Error updating tablets from 525198593c8e47afaee54e74267b87e3: Illegal state: replica b303b7df086340a0b5ad6373940ae750 is not leader of this config: current role FOLLOWER. Tablet report was: is_incremental: true updated_tablets { tablet_id: "e06daadd346d45969ad5efcf03bc0870" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "ed109b49f20642f988161a3df7872419" state: RUNNING consensus_state { current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "038ea1ea231e490ba27ecd5430883058" state: RUNNING consensus_state { current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "612987f3e4ea4a02932ceefa0d740425" state: RUNNING consensus_state { current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: UNKNOWN } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { } } updated_tablets { tablet_id: "a2d68eccaaa64511a9ae36f2b07a7b59" state: RUNNING consensus_state { current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: UNKNOWN } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { } } updated_tablets { tablet_id: "797d6cdf36704c98aa106aac87a945ca" state: RUNNING consensus_state { current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "5d545ac27a7f49faace0655713405cea" state: RUNNING consensus_state { current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 26732 live_row_count: 0 } } updated_tablets { tablet_id: "0b8989d340fa4ae992ef97e226d41110" state: RUNNING consensus_state { current_term: 1 leader_uuid: "90d6c2c58bb54e0eafb365625ac33871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "f40c66314342453b898da3ac0e7e86bf" state: RUNNING consensus_state { current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 26810 live_row_count: 0 } } updated_tablets { tablet_id: "30a56fb2d941470590e17f37b90aef26" state: RUNNING consensus_state { current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 26841 live_row_count: 0 } } updated_tablets { tablet_id: "fc406eab968745d893f1a8538ff61f37" state: RUNNING consensus_state { current_term: 2 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 26841 live_row_count: 0 } } updated_tablets { tablet_id: "59a46bc865cc48df8e98598764fd1b54" state: RUNNING consensus_state { current_term: 1 leader_uuid: "90d6c2c58bb54e0eafb365625ac33871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "6e2fb05f72b743e2a02ed36b63f80b23" state: RUNNING consensus_state { current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 26841 live_row_count: 0 } } updated_tablets { tablet_id: "97de018391f84b0ebd3130861681b7b5" state: RUNNING consensus_state { current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "a83bae91c29b463aa78412ce2e152ade" state: RUNNING consensus_state { current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 26810 live_row_count: 0 } } updated_tablets { tablet_id: "a52d6749ff834b62a8e20a2f45935240" state: RUNNING consensus_state { current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 26841 live_row_count: 0 } } updated_tablets { tablet_id: "d56255ca806d4e2cac09fbb5e1b1efbd" state: RUNNING consensus_state { current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: UNKNOWN } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { } } updated_tablets { tablet_id: "df045d80d8ce4ff194a6d2039a44bce6" state: RUNNING consensus_state { current_term: 1 leader_uuid: "819f9e3412e54d44abb11d8b3ca877c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "f464eabfbc0945fa8a5583f00c1dab40" state: RUNNING consensus_state { current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27063 live_row_count: 0 } } updated_tablets { tablet_id: "719188f639724851a5fc2b37cb60fdbb" state: RUNNING consensus_state { current_term: 2 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 26841 live_row_count: 0 } } updated_tablets { tablet_id: "4bb9b429af92446bbe13894d5dfe4c47" state: RUNNING consensus_state { current_term: 1 leader_uuid: "819f9e3412e54d44abb11d8b3ca877c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } } } schema_version: 2 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "15704785f56e43a3af69f181c75877ce" state: RUNNING consensus_state { current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27063 live_row_count: 0 } } updated_tablets { tablet_id: "044832851f804208b0a371eb0857dbf3" state: RUNNING consensus_state { current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27063 live_row_count: 0 } } updated_tablets { tablet_id: "ddfbab181b1f474192c664cd0316037f" state: RUNNING consensus_state { current_term: 1 leader_uuid: "90d6c2c58bb54e0eafb365625ac33871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "92cd8eccacf34ab7a66b3bbc69df3697" state: RUNNING consensus_state { current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } } } schema_version: 2 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27063 live_row_count: 0 } } updated_tablets { tablet_id: "97f4e061d3b94ee9982410b7ce135453" state: RUNNING consensus_state { current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 26841 live_row_count: 0 } } updated_tablets { tablet_id: "68981913851243a7a1875fc99efb3c49" state: RUNNING consensus_state { current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: UNKNOWN } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 26732 live_row_count: 0 } } updated_tablets { tablet_id: "0304df55431e42e1a999ed17dc35e6c0" state: RUNNING consensus_state { current_term: 1 leader_uuid: "90d6c2c58bb54e0eafb365625ac33871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "aad96f2507974023aaa988e6950feb7a" state: RUNNING consensus_state { current_term: 1 leader_uuid: "819f9e3412e54d44abb11d8b3ca877c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } } } schema_version: 2 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "ce7d6088fc884a248918d2348adcaccf" state: RUNNING consensus_state { current_term: 1 leader_uuid: "90d6c2c58bb54e0eafb365625ac33871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "f2b6812e8c1d44529d0284be490df648" state: RUNNING consensus_state { current_term: 1 leader_uuid: "819f9e3412e54d44abb11d8b3ca877c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "6448eea21ab2422c97da2494658ded3f" state: RUNNING consensus_state { current_term: 1 leader_uuid: "90d6c2c58bb54e0eafb365625ac33871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } } } schema_version: 2 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "d2a7b2bee5c1440fb705b172c18a9247" state: RUNNING consensus_state { current_term: 1 leader_uuid: "90d6c2c58bb54e0eafb365625ac33871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "967f4d5b944543d49e87645d069b5b55" state: RUNNING consensus_state { current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 26841 live_row_count: 0 } } updated_tablets { tablet_id: "71bd00f7d8ac41839f36fd73997631ad" state: RUNNING consensus_state { current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 26841 live_row_count: 0 } } updated_tablets { tablet_id: "8b4bc535c42e43889401f14b6d59659e" state: RUNNING consensus_state { current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27063 live_row_count: 0 } } updated_tablets { tablet_id: "edc69b66bea54625a4b15867b75986b7" state: RUNNING consensus_state { current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27063 live_row_count: 0 } } updated_tablets { tablet_id: "55b68586ef8b45439a7dd8dcd2efcd79" state: RUNNING consensus_state { current_term: 1 leader_uuid: "90d6c2c58bb54e0eafb365625ac33871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "2e17f619ed2745078541cc1c924aff96" state: RUNNING consensus_state { current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "9bfd8fd6d2104fb4a6484cb8c98981bd" state: RUNN
W20250114 02:54:40.567584 1042 consensus_peers.cc:487] T e3af8f1299f24b08b2d25811942aaecc P 90d6c2c58bb54e0eafb365625ac33871 -> Peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Couldn't send request to peer 819f9e3412e54d44abb11d8b3ca877c2. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.65:39397: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250114 02:54:40.567771 1207 ts_tablet_manager.cc:1428] T 992bb16197b74be5b1dc884f29d9c240 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.053s user 0.005s sys 0.000s
I20250114 02:54:40.571801 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_bbdaf710a91a429f9dd60f183ceea9f9 [id=4507c028bd074fd0a1d0231ea671b355]
I20250114 02:54:40.577265 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_972e696cae0e4f05a434e5375f6767c5 [id=456f726bff3d46e3b454f9b10e84c3c8]
I20250114 02:54:40.577584 2662 raft_consensus.cc:491] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 687d4ff6d3cb42cf90a965070b0d8a70)
I20250114 02:54:40.578032 2662 raft_consensus.cc:513] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b303b7df086340a0b5ad6373940ae750" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 45367 } } peers { permanent_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } } peers { permanent_uuid: "87b3db09ac87436e9a60d6de5fedaff5" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } }
I20250114 02:54:40.578850 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_f914e27e6a2e4144acdbda801011283b [id=45d8444e54064c709c13103963c1e9ba]
I20250114 02:54:40.579870 2662 leader_election.cc:290] T 00000000000000000000000000000000 P b303b7df086340a0b5ad6373940ae750 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 687d4ff6d3cb42cf90a965070b0d8a70 (127.0.107.125:46837), 87b3db09ac87436e9a60d6de5fedaff5 (127.0.107.124:39105)
W20250114 02:54:40.581615 466 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.66:50663 (request call id 59) took 2664 ms (client timeout 1000 ms). Trace:
I20250114 02:54:40.582031 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b303b7df086340a0b5ad6373940ae750" candidate_term: 3 candidate_status { last_received { term: 2 index: 441 } } ignore_live_leader: false dest_uuid: "87b3db09ac87436e9a60d6de5fedaff5" is_pre_election: true
W20250114 02:54:40.581915 466 rpcz_store.cc:269] 0114 02:54:37.917082 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:38.777953 (+860871us) service_pool.cc:225] Handling call
0114 02:54:40.387299 (+1609346us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:40.387319 (+ 20us) write_op.cc:454] Released schema lock
0114 02:54:40.581552 (+194233us) inbound_call.cc:173] Queueing failure response
Related trace 'op':
0114 02:54:40.387088 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica b303b7df086340a0b5ad6373940ae750 is not leader of this config: current role FOLLOWER)
0114 02:54:40.387121 (+ 33us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:40.387168 (+ 47us) write_op.cc:454] Released schema lock
0114 02:54:40.387187 (+ 19us) write_op.cc:335] FINISH: Op aborted
Metrics: {"reported_tablets":50,"child_traces":[["op",{}]]}
I20250114 02:54:40.595348 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_123095518d164d3db7fe3bd50af9dbd4 [id=4744eda447ca4785a9c065aca94135e9]
I20250114 02:54:40.597735 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_b4dc2961391c43fcb08fa062d4e45b3c [id=491b34b03ce14bd9be16e04915468928]
I20250114 02:54:40.608673 1207 tablet_bootstrap.cc:492] T ddad7584337345109ce102702008ab37 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:40.615587 1207 tablet_bootstrap.cc:654] T ddad7584337345109ce102702008ab37 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
I20250114 02:54:40.606285 2625 ts_tablet_manager.cc:1918] T 99b940d872ce4781907517c40fe08781 P 90d6c2c58bb54e0eafb365625ac33871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250114 02:54:40.616289 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_395ca40035c8410ca1ea6b0e1d54bf37 [id=4ce0efd1198d4af28694b13aab4299fd]
I20250114 02:54:40.617738 2625 log.cc:1198] T 99b940d872ce4781907517c40fe08781 P 90d6c2c58bb54e0eafb365625ac33871: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/99b940d872ce4781907517c40fe08781
I20250114 02:54:40.623742 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_3c0c8ba6eec34be4a6ea569d66038106 [id=4d2f7fabf8a44155a586a2b79a8fca53]
I20250114 02:54:40.625524 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_0518d8bed7304c86aff32e05e6f07c0d [id=515a2fe3f1e84e82bccc02ef5cecef1b]
I20250114 02:54:40.626108 2625 ts_tablet_manager.cc:1939] T 99b940d872ce4781907517c40fe08781 P 90d6c2c58bb54e0eafb365625ac33871: Deleting consensus metadata
I20250114 02:54:40.642242 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_58e014a2cfee480184ff3c66b1768cf6 [id=5745501ecd7b4cecaca3d35502a6cc40]
W20250114 02:54:40.646061 1042 consensus_peers.cc:487] T d2a7b2bee5c1440fb705b172c18a9247 P 90d6c2c58bb54e0eafb365625ac33871 -> Peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Couldn't send request to peer 819f9e3412e54d44abb11d8b3ca877c2. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.65:39397: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250114 02:54:40.653409 469 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 141cede72b8741a8b0615d7c6bb823bb (table default.Table_4364d1df16d149d2ba10469ce7288d4e [id=13fc996d911b430290e37a94dd44b1ab]). Expected version 1 got 0
I20250114 02:54:40.655474 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_e23da46545d64974b1f1563734b3c21c [id=5a3e47eb050545e09755d7364d0a30fe]
I20250114 02:54:40.655169 469 catalog_manager.cc:5526] T 2e17f619ed2745078541cc1c924aff96 P 525198593c8e47afaee54e74267b87e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65). New cstate: current_term: 1 leader_uuid: "819f9e3412e54d44abb11d8b3ca877c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } }
I20250114 02:54:40.665302 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_494541d472dc48d0bef33a634c1180fa [id=5ae72503278349c0bbe315174c896074]
I20250114 02:54:40.666018 2625 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 381) took 1791 ms. Trace:
I20250114 02:54:40.666347 2625 rpcz_store.cc:276] 0114 02:54:38.874115 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:38.874199 (+ 84us) service_pool.cc:225] Handling call
0114 02:54:40.479028 (+1604829us) ts_tablet_manager.cc:1145] Deleting tablet 99b940d872ce4781907517c40fe08781
0114 02:54:40.479066 (+ 38us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:40.530651 (+ 51585us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:40.603607 (+ 72956us) tablet_metadata.cc:641] Metadata flushed
0114 02:54:40.665960 (+ 62353us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1604332}
I20250114 02:54:40.667955 469 catalog_manager.cc:5526] T e7ca0bd1dbd542de9ee11e6c72522ae0 P 525198593c8e47afaee54e74267b87e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 525198593c8e47afaee54e74267b87e3 (127.0.107.66). New cstate: current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:40.692389 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_3652257c54c3434da1b93eb0481a0df7 [id=5e6aa799d6444bad968f29327ebace9b]
I20250114 02:54:40.693116 469 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet df045d80d8ce4ff194a6d2039a44bce6 (table default.Table_2befe573a994473abe0556ba7e713686 [id=da0258578bb34ae29551fc593bf7d0b5]). Expected version 1 got 0
I20250114 02:54:40.694722 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_fb66bdd04429412b9585afd33d5b8876 [id=5ff61b779680460098f62f3aa92ed309]
I20250114 02:54:40.694103 469 catalog_manager.cc:5526] T a83bae91c29b463aa78412ce2e152ade P 525198593c8e47afaee54e74267b87e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 525198593c8e47afaee54e74267b87e3 (127.0.107.66). New cstate: current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } }
I20250114 02:54:40.701050 469 catalog_manager.cc:5526] T 612987f3e4ea4a02932ceefa0d740425 P 525198593c8e47afaee54e74267b87e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 525198593c8e47afaee54e74267b87e3 (127.0.107.66). New cstate: current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } }
I20250114 02:54:40.702170 2625 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 383) took 1826 ms. Trace:
I20250114 02:54:40.703200 1207 tablet_bootstrap.cc:492] T ddad7584337345109ce102702008ab37 P 90d6c2c58bb54e0eafb365625ac33871: No bootstrap required, opened a new log
I20250114 02:54:40.702533 2625 rpcz_store.cc:276] 0114 02:54:38.875481 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:38.875572 (+ 91us) service_pool.cc:225] Handling call
0114 02:54:40.677244 (+1801672us) ts_tablet_manager.cc:1145] Deleting tablet d05089eae2b74f9fb23a226d358a59da
0114 02:54:40.701864 (+ 24620us) spinlock_profiling.cc:243] Waited 3.01 ms on lock 0x7b5800001898. stack: 00007fc2418dcf63 00007fc2418dce60 00007fc241453584 00000000004fcd24 00000000004fccb9 00007fc241752911 00007fc2418d1991 00007fc2418d1fc2 00007fc24a564999 00007fc2497c95f8 00007fc24988d3e9 00007fc24988d9e3 00007fc2498bdca0 00007fc2498a4e6d 00007fc2498a4e19 00007fc2498a4da9
0114 02:54:40.701917 (+ 53us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:40.702113 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6931200,"tablet-delete.queue_time_us":1800503}
I20250114 02:54:40.703600 1207 ts_tablet_manager.cc:1397] T ddad7584337345109ce102702008ab37 P 90d6c2c58bb54e0eafb365625ac33871: Time spent bootstrapping tablet: real 0.095s user 0.011s sys 0.000s
I20250114 02:54:40.704998 469 catalog_manager.cc:5526] T 68981913851243a7a1875fc99efb3c49 P 525198593c8e47afaee54e74267b87e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 525198593c8e47afaee54e74267b87e3 (127.0.107.66). New cstate: current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } }
I20250114 02:54:40.706701 1207 raft_consensus.cc:357] T ddad7584337345109ce102702008ab37 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:40.707312 1207 raft_consensus.cc:383] T ddad7584337345109ce102702008ab37 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:54:40.707585 1207 raft_consensus.cc:738] T ddad7584337345109ce102702008ab37 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d6c2c58bb54e0eafb365625ac33871, State: Initialized, Role: FOLLOWER
I20250114 02:54:40.712647 469 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 68981913851243a7a1875fc99efb3c49 (table default.Table_5e2861a7715a47039e64f1a717c57997 [id=683a61c5af09453e8027e9cb5b5b80a7]). Expected version 1 got 0
I20250114 02:54:40.718931 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_0e0710e3f43e4e859ad32662704ee41c [id=66082998119e4c43a666fb43087fc622]
I20250114 02:54:40.719607 469 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 719188f639724851a5fc2b37cb60fdbb (table default.Table_8f8a18d7868c44bbb4d66a0692b08186 [id=e6292728826f437cb75c524a8aef2e87]). Expected version 1 got 0
I20250114 02:54:40.721275 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_5e2861a7715a47039e64f1a717c57997 [id=683a61c5af09453e8027e9cb5b5b80a7]
I20250114 02:54:40.722317 469 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet fc406eab968745d893f1a8538ff61f37 (table default.Table_7367bdf975b243aab9651828ad630606 [id=e1ca4114e55d48839dfa2f76624a4234]). Expected version 1 got 0
I20250114 02:54:40.722895 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_1a9735a294c2436e84ec9db5e0aa44e4 [id=6922b5f98cc14e1d8a7e745211aa7b45]
I20250114 02:54:40.723556 469 catalog_manager.cc:5526] T be7aad41ccdd4ef9bcfe8a985155708f P 525198593c8e47afaee54e74267b87e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67). New cstate: current_term: 1 leader_uuid: "90d6c2c58bb54e0eafb365625ac33871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } }
I20250114 02:54:40.734375 469 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 30a56fb2d941470590e17f37b90aef26 (table default.Table_0e0710e3f43e4e859ad32662704ee41c [id=66082998119e4c43a666fb43087fc622]). Expected version 1 got 0
I20250114 02:54:40.735952 469 catalog_manager.cc:5526] T 038ea1ea231e490ba27ecd5430883058 P 525198593c8e47afaee54e74267b87e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 525198593c8e47afaee54e74267b87e3 (127.0.107.66). New cstate: current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: UNKNOWN } } }
I20250114 02:54:40.742564 2625 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53876 (request call id 382) took 1867 ms. Trace:
I20250114 02:54:40.742980 2625 rpcz_store.cc:276] 0114 02:54:38.875239 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:38.875636 (+ 397us) service_pool.cc:225] Handling call
0114 02:54:40.726454 (+1850818us) ts_tablet_manager.cc:1145] Deleting tablet 6e858a8449444ef999b9bbef612249d3
0114 02:54:40.727983 (+ 1529us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0114 02:54:40.742409 (+ 14426us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1850090}
I20250114 02:54:40.743528 1207 consensus_queue.cc:260] T ddad7584337345109ce102702008ab37 P 90d6c2c58bb54e0eafb365625ac33871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
W20250114 02:54:40.747737 1042 consensus_peers.cc:487] T 141cede72b8741a8b0615d7c6bb823bb P 90d6c2c58bb54e0eafb365625ac33871 -> Peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Couldn't send request to peer 819f9e3412e54d44abb11d8b3ca877c2. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.65:39397: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250114 02:54:40.732983 458 catalog_manager.cc:4872] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): tablet 99b940d872ce4781907517c40fe08781 (table default.Table_62251807af5845a89e89f8b2e003928f [id=e3418787ef0a4e22979fb41d4f94ebb0]) successfully deleted
W20250114 02:54:40.751684 458 catalog_manager.cc:4839] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet d05089eae2b74f9fb23a226d358a59da because the tablet was not found. No further retry: Not found: Tablet not found: d05089eae2b74f9fb23a226d358a59da
W20250114 02:54:40.752302 458 catalog_manager.cc:4839] TS 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67:45749): delete failed for tablet 6e858a8449444ef999b9bbef612249d3 because the tablet was not found. No further retry: Not found: Tablet not found: 6e858a8449444ef999b9bbef612249d3
I20250114 02:54:40.755126 2510 raft_consensus.cc:491] T ddad7584337345109ce102702008ab37 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:40.755622 2510 raft_consensus.cc:513] T ddad7584337345109ce102702008ab37 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:40.756510 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_d0eabdfdf4214a32b3afc2c70c63eb9c [id=6ce308f2d2dd4b53bed4a3c6b973c7e1]
I20250114 02:54:40.757494 2510 leader_election.cc:290] T ddad7584337345109ce102702008ab37 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
W20250114 02:54:40.761628 1042 leader_election.cc:336] T ddad7584337345109ce102702008ab37 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Network error: Client connection negotiation failed: client connection to 127.0.107.65:39397: connect: Connection refused (error 111)
I20250114 02:54:40.765179 1207 ts_tablet_manager.cc:1428] T ddad7584337345109ce102702008ab37 P 90d6c2c58bb54e0eafb365625ac33871: Time spent starting tablet: real 0.061s user 0.006s sys 0.001s
I20250114 02:54:40.771078 2625 tablet_replica.cc:331] T 59a46bc865cc48df8e98598764fd1b54 P 90d6c2c58bb54e0eafb365625ac33871: stopping tablet replica
I20250114 02:54:40.771873 2625 raft_consensus.cc:2238] T 59a46bc865cc48df8e98598764fd1b54 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 LEADER]: Raft consensus shutting down.
I20250114 02:54:40.772614 2625 raft_consensus.cc:2267] T 59a46bc865cc48df8e98598764fd1b54 P 90d6c2c58bb54e0eafb365625ac33871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:54:40.775187 2625 ts_tablet_manager.cc:1905] T 59a46bc865cc48df8e98598764fd1b54 P 90d6c2c58bb54e0eafb365625ac33871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:54:40.780934 1207 tablet_bootstrap.cc:492] T dbb1d0851221497ca2b41844526b4502 P 90d6c2c58bb54e0eafb365625ac33871: Bootstrap starting.
I20250114 02:54:40.788528 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_26478578c48e45af84e6b020c897a5f2 [id=77e4cc4023c141099fa2555667dc7ed3]
I20250114 02:54:40.789739 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_ae6b6fdfc1834cfb8089c860aedcc964 [id=7da003630a7a4e269d54fb2af8a3d4b5]
I20250114 02:54:40.790858 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_96b6a43fc44a43ce9f6485f0e8daae5d [id=7e19cb37c53e4636941205a2d542984b]
I20250114 02:54:40.791989 469 catalog_manager.cc:5526] T e06daadd346d45969ad5efcf03bc0870 P 525198593c8e47afaee54e74267b87e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67). New cstate: current_term: 1 leader_uuid: "90d6c2c58bb54e0eafb365625ac33871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } }
I20250114 02:54:40.793509 469 catalog_manager.cc:5526] T 0b8989d340fa4ae992ef97e226d41110 P 525198593c8e47afaee54e74267b87e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67). New cstate: current_term: 1 leader_uuid: "90d6c2c58bb54e0eafb365625ac33871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } }
I20250114 02:54:40.794873 469 catalog_manager.cc:5526] T 797d6cdf36704c98aa106aac87a945ca P 525198593c8e47afaee54e74267b87e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 525198593c8e47afaee54e74267b87e3 (127.0.107.66). New cstate: current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: UNKNOWN } } }
W20250114 02:54:40.798468 1042 connection.cc:537] server connection from 127.0.107.66:32905 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250114 02:54:40.804839 2510 raft_consensus.cc:491] T 992bb16197b74be5b1dc884f29d9c240 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:54:40.805342 2510 raft_consensus.cc:513] T 992bb16197b74be5b1dc884f29d9c240 P 90d6c2c58bb54e0eafb365625ac33871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } }
I20250114 02:54:40.807052 1207 tablet_bootstrap.cc:654] T dbb1d0851221497ca2b41844526b4502 P 90d6c2c58bb54e0eafb365625ac33871: Neither blocks nor log segments found. Creating new log.
W20250114 02:54:40.807639 1042 consensus_peers.cc:487] T 6448eea21ab2422c97da2494658ded3f P 90d6c2c58bb54e0eafb365625ac33871 -> Peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Couldn't send request to peer 819f9e3412e54d44abb11d8b3ca877c2. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.65:39397: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250114 02:54:40.811996 2510 leader_election.cc:290] T 992bb16197b74be5b1dc884f29d9c240 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397), 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037)
W20250114 02:54:40.812579 1042 leader_election.cc:336] T 992bb16197b74be5b1dc884f29d9c240 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 819f9e3412e54d44abb11d8b3ca877c2 (127.0.107.65:39397): Network error: Client connection negotiation failed: client connection to 127.0.107.65:39397: connect: Connection refused (error 111)
W20250114 02:54:40.813277 1040 connection.cc:537] client connection to 127.0.107.66:44037 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250114 02:54:40.819357 456 connection.cc:537] server connection from 127.0.107.66:50663 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250114 02:54:40.823133 2704 negotiation.cc:337] Failed RPC negotiation. Trace:
0114 02:54:40.816640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.107.66:44037 (local address 127.0.107.67:45325)
0114 02:54:40.818200 (+ 1560us) negotiation.cc:107] Waiting for socket to connect
0114 02:54:40.818255 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0114 02:54:40.818649 (+ 394us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0114 02:54:40.822506 (+ 3857us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.107.66:44037: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1062}
W20250114 02:54:40.823746 1040 consensus_peers.cc:487] T 45301f8823ed4ec494e792d41f14f71f P 90d6c2c58bb54e0eafb365625ac33871 -> Peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Couldn't send request to peer 525198593c8e47afaee54e74267b87e3. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.66:44037: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20250114 02:54:40.824170 1040 consensus_peers.cc:487] T 32dd43e8a3544dbaa6e2fbb4b4c1dc28 P 90d6c2c58bb54e0eafb365625ac33871 -> Peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Couldn't send request to peer 525198593c8e47afaee54e74267b87e3. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.66:44037: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20250114 02:54:40.824714 1040 consensus_peers.cc:487] T be7aad41ccdd4ef9bcfe8a985155708f P 90d6c2c58bb54e0eafb365625ac33871 -> Peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Couldn't send request to peer 525198593c8e47afaee54e74267b87e3. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.66:44037: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
I20250114 02:54:40.796121 469 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 797d6cdf36704c98aa106aac87a945ca (table default.Table_5a9480c6474848d6b400e7dd274d15b0 [id=a17ad30ef32f4cbcb22b5a7d1c7096ff]). Expected version 1 got 0
W20250114 02:54:40.825181 1040 leader_election.cc:336] T ddad7584337345109ce102702008ab37 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Network error: Client connection negotiation failed: client connection to 127.0.107.66:44037: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250114 02:54:40.825587 1040 leader_election.cc:304] T ddad7584337345109ce102702008ab37 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90d6c2c58bb54e0eafb365625ac33871; no voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2
W20250114 02:54:40.826308 1040 consensus_peers.cc:487] T 6448eea21ab2422c97da2494658ded3f P 90d6c2c58bb54e0eafb365625ac33871 -> Peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Couldn't send request to peer 525198593c8e47afaee54e74267b87e3. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.66:44037: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20250114 02:54:40.826706 1040 consensus_peers.cc:487] T 141cede72b8741a8b0615d7c6bb823bb P 90d6c2c58bb54e0eafb365625ac33871 -> Peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Couldn't send request to peer 525198593c8e47afaee54e74267b87e3. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.66:44037: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20250114 02:54:40.827042 1040 leader_election.cc:336] T 992bb16197b74be5b1dc884f29d9c240 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037): Network error: Client connection negotiation failed: client connection to 127.0.107.66:44037: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250114 02:54:40.827373 1040 leader_election.cc:304] T 992bb16197b74be5b1dc884f29d9c240 P 90d6c2c58bb54e0eafb365625ac33871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90d6c2c58bb54e0eafb365625ac33871; no voters: 525198593c8e47afaee54e74267b87e3, 819f9e3412e54d44abb11d8b3ca877c2
I20250114 02:54:40.827728 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_23e9951e382b44538252984608700dcc [id=804e11d5d5e64f2897ff2265ff77427d]
I20250114 02:54:40.829083 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_a573620245b049398a53d6cb8799ca39 [id=81e2d41193404732b5db92e1357b8c81]
W20250114 02:54:40.829257 2707 negotiation.cc:337] Failed RPC negotiation. Trace:
0114 02:54:40.820978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.107.66:44037 (local address 127.0.0.1:50996)
0114 02:54:40.821721 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0114 02:54:40.821757 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0114 02:54:40.822060 (+ 303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0114 02:54:40.828402 (+ 6342us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.107.66:44037: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":215}
I20250114 02:54:40.830873 429 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrCUXNq/build/tsan/bin/kudu with pid 1017
I20250114 02:54:40.832258 469 catalog_manager.cc:5526] T 5d545ac27a7f49faace0655713405cea P 525198593c8e47afaee54e74267b87e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 525198593c8e47afaee54e74267b87e3 (127.0.107.66). New cstate: current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } }
I20250114 02:54:40.833498 469 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet 5d545ac27a7f49faace0655713405cea (table default.Table_b4dc2961391c43fcb08fa062d4e45b3c [id=491b34b03ce14bd9be16e04915468928]). Expected version 1 got 0
I20250114 02:54:40.834146 469 catalog_manager.cc:5526] T a2d68eccaaa64511a9ae36f2b07a7b59 P 525198593c8e47afaee54e74267b87e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 525198593c8e47afaee54e74267b87e3 (127.0.107.66). New cstate: current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } }
I20250114 02:54:40.836733 2687 raft_consensus.cc:491] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 687d4ff6d3cb42cf90a965070b0d8a70)
I20250114 02:54:40.837157 2687 raft_consensus.cc:513] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b303b7df086340a0b5ad6373940ae750" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 45367 } } peers { permanent_uuid: "687d4ff6d3cb42cf90a965070b0d8a70" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 46837 } } peers { permanent_uuid: "87b3db09ac87436e9a60d6de5fedaff5" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 39105 } }
W20250114 02:54:40.838372 456 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 797d6cdf36704c98aa106aac87a945ca (table default.Table_5a9480c6474848d6b400e7dd274d15b0 [id=a17ad30ef32f4cbcb22b5a7d1c7096ff], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 797d6cdf36704c98aa106aac87a945ca consensus state has no leader
I20250114 02:54:40.838354 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_bacdf22532c84100a69a38cfa0ad07e5 [id=86304f6b48fd421cb953070ae7c9b0a2]
I20250114 02:54:40.838995 2687 leader_election.cc:290] T 00000000000000000000000000000000 P 87b3db09ac87436e9a60d6de5fedaff5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b303b7df086340a0b5ad6373940ae750 (127.0.107.126:45367), 687d4ff6d3cb42cf90a965070b0d8a70 (127.0.107.125:46837)
I20250114 02:54:40.837620 469 catalog_manager.cc:5526] T ed109b49f20642f988161a3df7872419 P 525198593c8e47afaee54e74267b87e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90d6c2c58bb54e0eafb365625ac33871 (127.0.107.67). New cstate: current_term: 1 leader_uuid: "90d6c2c58bb54e0eafb365625ac33871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } }
I20250114 02:54:40.840358 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_d21051d46c4043488adb2d04cfb8cead [id=87438984bbfd4f89b81a4550d30130f8]
I20250114 02:54:40.840343 481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "87b3db09ac87436e9a60d6de5fedaff5" candidate_term: 3 candidate_status { last_received { term: 2 index: 441 } } ignore_live_leader: false dest_uuid: "b303b7df086340a0b5ad6373940ae750" is_pre_election: true
I20250114 02:54:40.842298 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_c8f09aee14204d5ea1c506f230fb81b6 [id=8d9132464cb64cc096578290ede567f7]
I20250114 02:54:40.841919 469 catalog_manager.cc:5618] TS 525198593c8e47afaee54e74267b87e3 (127.0.107.66:44037) does not have the latest schema for tablet d2a7b2bee5c1440fb705b172c18a9247 (table default.Table_3c0c8ba6eec34be4a6ea569d66038106 [id=4d2f7fabf8a44155a586a2b79a8fca53]). Expected version 1 got 0
I20250114 02:54:40.843631 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_4126dce6d1614249887aee151c0b1085 [id=8d94afa1935a428ba36c8171805cfa9f]
I20250114 02:54:40.881722 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_1d46f092cec5487fbf47ab3d83aaeb87 [id=903117c8ff5f4ecf9913fc4fe6280465]
I20250114 02:54:40.883095 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_c50c01c8bc7e44a3bd139bc573c70122 [id=9171c778af084ef2bb6032e09a096769]
E20250114 02:54:40.871712 469 catalog_manager.cc:5681] Error updating tablets from 525198593c8e47afaee54e74267b87e3: Illegal state: replica b303b7df086340a0b5ad6373940ae750 is not leader of this config: current role FOLLOWER. Tablet report was: is_incremental: true updated_tablets { tablet_id: "2d4eaf90be6d497d8a3c4d9ac418f4e9" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "e7ca0bd1dbd542de9ee11e6c72522ae0" state: RUNNING consensus_state { current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: UNKNOWN } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { } } updated_tablets { tablet_id: "e06daadd346d45969ad5efcf03bc0870" state: RUNNING consensus_state { current_term: 1 leader_uuid: "90d6c2c58bb54e0eafb365625ac33871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "ed109b49f20642f988161a3df7872419" state: RUNNING consensus_state { current_term: 1 leader_uuid: "90d6c2c58bb54e0eafb365625ac33871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "038ea1ea231e490ba27ecd5430883058" state: RUNNING consensus_state { current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: UNKNOWN } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { } } updated_tablets { tablet_id: "612987f3e4ea4a02932ceefa0d740425" state: RUNNING consensus_state { current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { } } updated_tablets { tablet_id: "a2d68eccaaa64511a9ae36f2b07a7b59" state: RUNNING consensus_state { current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { } } updated_tablets { tablet_id: "797d6cdf36704c98aa106aac87a945ca" state: RUNNING consensus_state { current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: UNKNOWN } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { } } updated_tablets { tablet_id: "5d545ac27a7f49faace0655713405cea" state: RUNNING consensus_state { current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 26732 live_row_count: 0 } } updated_tablets { tablet_id: "0b8989d340fa4ae992ef97e226d41110" state: RUNNING consensus_state { current_term: 1 leader_uuid: "90d6c2c58bb54e0eafb365625ac33871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "f40c66314342453b898da3ac0e7e86bf" state: RUNNING consensus_state { current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 26810 live_row_count: 0 } } updated_tablets { tablet_id: "30a56fb2d941470590e17f37b90aef26" state: RUNNING consensus_state { current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 26841 live_row_count: 0 } } updated_tablets { tablet_id: "fc406eab968745d893f1a8538ff61f37" state: RUNNING consensus_state { current_term: 2 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 26841 live_row_count: 0 } } updated_tablets { tablet_id: "59a46bc865cc48df8e98598764fd1b54" state: RUNNING consensus_state { current_term: 1 leader_uuid: "90d6c2c58bb54e0eafb365625ac33871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "6e2fb05f72b743e2a02ed36b63f80b23" state: RUNNING consensus_state { current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 26841 live_row_count: 0 } } updated_tablets { tablet_id: "97de018391f84b0ebd3130861681b7b5" state: RUNNING consensus_state { current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "a83bae91c29b463aa78412ce2e152ade" state: RUNNING consensus_state { current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 26810 live_row_count: 0 } } updated_tablets { tablet_id: "a52d6749ff834b62a8e20a2f45935240" state: RUNNING consensus_state { current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 26841 live_row_count: 0 } } updated_tablets { tablet_id: "d56255ca806d4e2cac09fbb5e1b1efbd" state: RUNNING consensus_state { current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: UNKNOWN } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { } } updated_tablets { tablet_id: "df045d80d8ce4ff194a6d2039a44bce6" state: RUNNING consensus_state { current_term: 1 leader_uuid: "819f9e3412e54d44abb11d8b3ca877c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "f464eabfbc0945fa8a5583f00c1dab40" state: RUNNING consensus_state { current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27063 live_row_count: 0 } } updated_tablets { tablet_id: "719188f639724851a5fc2b37cb60fdbb" state: RUNNING consensus_state { current_term: 2 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 26841 live_row_count: 0 } } updated_tablets { tablet_id: "4bb9b429af92446bbe13894d5dfe4c47" state: RUNNING consensus_state { current_term: 1 leader_uuid: "819f9e3412e54d44abb11d8b3ca877c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } } } schema_version: 2 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "15704785f56e43a3af69f181c75877ce" state: RUNNING consensus_state { current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27063 live_row_count: 0 } } updated_tablets { tablet_id: "044832851f804208b0a371eb0857dbf3" state: RUNNING consensus_state { current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27063 live_row_count: 0 } } updated_tablets { tablet_id: "ddfbab181b1f474192c664cd0316037f" state: RUNNING consensus_state { current_term: 1 leader_uuid: "90d6c2c58bb54e0eafb365625ac33871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "92cd8eccacf34ab7a66b3bbc69df3697" state: RUNNING consensus_state { current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } } } schema_version: 2 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27063 live_row_count: 0 } } updated_tablets { tablet_id: "97f4e061d3b94ee9982410b7ce135453" state: RUNNING consensus_state { current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 26841 live_row_count: 0 } } updated_tablets { tablet_id: "68981913851243a7a1875fc99efb3c49" state: RUNNING consensus_state { current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 26732 live_row_count: 0 } } updated_tablets { tablet_id: "0304df55431e42e1a999ed17dc35e6c0" state: RUNNING consensus_state { current_term: 1 leader_uuid: "90d6c2c58bb54e0eafb365625ac33871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "aad96f2507974023aaa988e6950feb7a" state: RUNNING consensus_state { current_term: 1 leader_uuid: "819f9e3412e54d44abb11d8b3ca877c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } } } schema_version: 2 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "ce7d6088fc884a248918d2348adcaccf" state: RUNNING consensus_state { current_term: 1 leader_uuid: "90d6c2c58bb54e0eafb365625ac33871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "f2b6812e8c1d44529d0284be490df648" state: RUNNING consensus_state { current_term: 1 leader_uuid: "819f9e3412e54d44abb11d8b3ca877c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "6448eea21ab2422c97da2494658ded3f" state: RUNNING consensus_state { current_term: 1 leader_uuid: "90d6c2c58bb54e0eafb365625ac33871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } } } schema_version: 2 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "d2a7b2bee5c1440fb705b172c18a9247" state: RUNNING consensus_state { current_term: 1 leader_uuid: "90d6c2c58bb54e0eafb365625ac33871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "967f4d5b944543d49e87645d069b5b55" state: RUNNING consensus_state { current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 26841 live_row_count: 0 } } updated_tablets { tablet_id: "71bd00f7d8ac41839f36fd73997631ad" state: RUNNING consensus_state { current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 26841 live_row_count: 0 } } updated_tablets { tablet_id: "8b4bc535c42e43889401f14b6d59659e" state: RUNNING consensus_state { current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819f9e3412e54d44abb11d8b3ca877c2" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 39397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d6c2c58bb54e0eafb365625ac33871" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 44037 } health_report { overall_health: HEALTHY } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27063 live_row_count: 0 } } updated_tablets { tablet_id: "edc69b66bea54625a4b15867b75986b7" state: RUNNING consensus_state { current_term: 1 leader_uuid: "525198593c8e47afaee54e74267b87e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "525198593c8e47afaee54e74267b87e3" member_type: VOTER last_known_addr { host: "127.0.
I20250114 02:54:40.884449 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_60450873174a4c80aa7e2122343d82be [id=9299683349354f879847b7c9d93f6072]
I20250114 02:54:40.901479 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_dda4f31205ab46c7a3af70bbb8037ec8 [id=94052a8a70044fab9b2753c326ad23d7]
I20250114 02:54:40.904178 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_63cdf90ef15a453c9acb19962262eb35 [id=985cc4adbb8a463db0b6a5a5dd49b0ca]
I20250114 02:54:40.905445 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_1ec80cfdcee64e98beccc30fec0afff8 [id=9cf0c5a5c3434e348bb3d159e9638924]
I20250114 02:54:40.906738 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_8d2e7b6869da423a949c59081ea3618d [id=9cf2e181b6bc44f3b96312c76ba6e646]
I20250114 02:54:40.908154 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_8826c1ea171946ab9bc9b3572bf349c4 [id=9d6566ee69c34ebc8b3b638d55babe88]
W20250114 02:54:40.945129 469 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.107.66:50663 (request call id 60) took 1814 ms (client timeout 1000 ms). Trace:
I20250114 02:54:40.945356 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_ada937b35c344a4b8d1c0a3d52fa23e2 [id=a02e469d3f7946ff863a84d5906e0009]
W20250114 02:54:40.945497 469 rpcz_store.cc:269] 0114 02:54:39.130964 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:54:39.131081 (+ 117us) service_pool.cc:225] Handling call
0114 02:54:40.871470 (+1740389us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:40.871489 (+ 19us) write_op.cc:454] Released schema lock
0114 02:54:40.945062 (+ 73573us) inbound_call.cc:173] Queueing failure response
Related trace 'op':
0114 02:54:40.871291 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica b303b7df086340a0b5ad6373940ae750 is not leader of this config: current role FOLLOWER)
0114 02:54:40.871323 (+ 32us) write_op.cc:489] Releasing partition, row and schema locks
0114 02:54:40.871341 (+ 18us) write_op.cc:454] Released schema lock
0114 02:54:40.871361 (+ 20us) write_op.cc:335] FINISH: Op aborted
Metrics: {"reported_tablets":53,"child_traces":[["op",{}]]}
I20250114 02:54:40.946559 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_23120ced31c744a0ba995f46d50223a3 [id=a0a99c63c698448499d5a85cb85740bb]
W20250114 02:54:40.947043 456 connection.cc:441] server connection from 127.0.107.66:50663 torn down before Call kudu.master.MasterService.TSHeartbeat from 127.0.107.66:50663 (request call id 60) could send its response
I20250114 02:54:40.947763 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_5a9480c6474848d6b400e7dd274d15b0 [id=a17ad30ef32f4cbcb22b5a7d1c7096ff]
I20250114 02:54:40.977523 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_b41debc5d26d4d5aa95793afabcec087 [id=a6825ddf945843ffaf897d6ca6b62923]
I20250114 02:54:40.996551 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_5940a20d29474d1a8dcc1d01e3599699 [id=ad58681da58346d9ad644c7429c37212]
I20250114 02:54:40.998268 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_529c38ed585343acb0f42ac60f171b59 [id=af3fec2e3d0243bfbed7aa314a9732a3]
I20250114 02:54:40.999380 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_dd50844b27ef42dda7cab6f276fac825 [id=af4dd8face234ca99bf7523004217964]
I20250114 02:54:41.015313 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_6bdcd457ba84407e8ec4b0fb6a1d04c0 [id=b0cedff1a9b44e249739d2dc53418a16]
I20250114 02:54:41.020526 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_1142a1a45945449ea16f5501c3adeb8b [id=b124b43245d14d7daebf8d33882c9124]
I20250114 02:54:41.023545 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_1c9b7896941e4289b1c12e20c210c2c1 [id=b3ee15b9f10f4d489edbde3af9034d25]
I20250114 02:54:41.037079 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_3f4f2aa33eae44a1ba3c243179bfd590 [id=b46c2d8e8a2b4c3b8d55c78e6c0dc401]
I20250114 02:54:41.039268 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_ec0eefb64458440ebd57a05a7ad0bb9a [id=b6874c22fe0d4151a7c8ea061f685938]
I20250114 02:54:41.060465 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_20609e5296574025b1567d6463933b17 [id=b825f6b5a125445aa2feddab78584d73]
I20250114 02:54:41.062909 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_174825b874fc410fba44e7a1335609d8 [id=bb1b35430a7b4fa5a8b65d9c68133093]
I20250114 02:54:41.064956 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_6e385f4df79b47cb9f1ada084a60abbf [id=be6d575261b944d6bc12cc5db002c77c]
I20250114 02:54:41.066259 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_382cf0d973d14925b8424879a1e6d7df [id=c0194382ca85417aa7ed838cef045a95]
I20250114 02:54:41.092859 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_2c307103a2e8474b9c7e18b5691f61d4 [id=c65db2976e214ac2b96b15b7a49487c1]
I20250114 02:54:41.095021 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_66f2263162f64250a0b49073abe4bd4c [id=c73ede1a5f664d8eb204fb9a012f8f7a]
I20250114 02:54:41.096256 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_161092dfc9464fae90944e56b7ba1740 [id=c805ce6c46c948049bb54209abf69d85]
I20250114 02:54:41.098222 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_cd7d049881524116bf642f02443f44ad [id=c9ca7d97b22041d482cdd4ec5467c865]
W20250114 02:54:41.101140 2707 negotiation.cc:337] Failed RPC negotiation. Trace:
0114 02:54:41.098766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.107.67:45749 (local address 127.0.0.1:43228)
0114 02:54:41.099755 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0114 02:54:41.099795 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0114 02:54:41.100021 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0114 02:54:41.100470 (+ 449us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.107.67:45749: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":391}
I20250114 02:54:41.102842 429 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrCUXNq/build/tsan/bin/kudu with pid 438
I20250114 02:54:41.135104 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_36be92916bc1421bb2c3261d44b3a6db [id=d233b7100cba40aebf374397a52ed778]
I20250114 02:54:41.138062 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_4fd891b63924429085d78f78cb177f0e [id=d36c71e7562b4ff1bc8be5cf2955000e]
I20250114 02:54:41.140118 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_24cfc302131c4035b4c8e0fdc03625fa [id=d772a301194a4e9ab90cabfc05104081]
I20250114 02:54:41.164085 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_1097f09fb10e441fb639f8d7aa12d487 [id=d8c9691818d042d7b49475fa009c4ca1]
I20250114 02:54:41.165457 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_2befe573a994473abe0556ba7e713686 [id=da0258578bb34ae29551fc593bf7d0b5]
I20250114 02:54:41.167546 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_403c29800daf456d93b3b60002fa6f96 [id=dcae1865f0b44cd8831a9fd0ab629415]
I20250114 02:54:41.168967 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_7367bdf975b243aab9651828ad630606 [id=e1ca4114e55d48839dfa2f76624a4234]
I20250114 02:54:41.170238 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_0931c459ae794d2091fcc0f51d031548 [id=e279c31bd63b40b9ade03a3b928bf153]
I20250114 02:54:41.171522 2673 catalog_manager.cc:670] Loaded metadata for table default.Table_62251807af5845a89e89f8b2e003928f [id=e3418787ef0a4e22979fb41d4f94ebb0]
I20250114 02:54:41.196537 429 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrCUXNq/build/tsan/bin/kudu with pid 508
W20250114 02:54:41.320538 2204 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.107.125:46837 (error 108)
I20250114 02:54:41.327777 429 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrCUXNq/build/tsan/bin/kudu with pid 1423
2025-01-14T02:54:41Z chronyd exiting
I20250114 02:54:41.384829 429 master-stress-test.cc:585] Tables created: 168
I20250114 02:54:41.385022 429 master-stress-test.cc:586] Tables altered: 71
I20250114 02:54:41.385185 429 master-stress-test.cc:587] Tables deleted: 59
I20250114 02:54:41.385313 429 master-stress-test.cc:588] Tablets replaced: 32
I20250114 02:54:41.385460 429 master-stress-test.cc:589] Masters restarted: 1
[ OK ] CatalogManagerConfigurations/MasterStressTest.Test/0 (47258 ms)
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/3
I20250114 02:54:41.475499 429 test_util.cc:274] Using random seed: -1363484781
2025-01-14T02:54:41Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-14T02:54:41Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskrCUXNq/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskrCUXNq/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/security.properties
2025-01-14 02:54:48: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskrCUXNq/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskrCUXNq/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20250114 02:54:56.196776 429 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrCUXNq/build/tsan/bin/kudu
/tmp/dist-test-taskrCUXNq/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.107.126:40235
--webserver_interface=127.0.107.126
--webserver_port=0
--builtin_ntp_servers=127.0.107.84:38241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.107.126:40235,127.0.107.125:38653,127.0.107.124:44431
--hive_metastore_uris=thrift://127.0.0.1:33943
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250114 02:54:56.494920 2952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 02:54:56.495525 2952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 02:54:56.495992 2952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 02:54:56.526201 2952 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20250114 02:54:56.526443 2952 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20250114 02:54:56.526715 2952 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250114 02:54:56.526961 2952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 02:54:56.527235 2952 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250114 02:54:56.527518 2952 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250114 02:54:56.562431 2952 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.107.84:38241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:33943
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.107.126:40235,127.0.107.125:38653,127.0.107.124:44431
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.107.126:40235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.107.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type FASTDEBUG
built by None at 14 Jan 2025 02:43:23 UTC on a633016ff5c9
build id 4168
TSAN enabled
I20250114 02:54:56.563782 2952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 02:54:56.565374 2952 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 02:54:56.582134 2961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 02:54:56.582904 2959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 02:54:56.584053 2958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 02:54:56.584872 2952 server_base.cc:1034] running on GCE node
I20250114 02:54:57.748023 2952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 02:54:57.750720 2952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 02:54:57.752071 2952 hybrid_clock.cc:648] HybridClock initialized: now 1736823297752023 us; error 63 us; skew 500 ppm
I20250114 02:54:57.752846 2952 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250114 02:54:57.764091 2952 webserver.cc:458] Webserver started at http://127.0.107.126:43931/ using document root <none> and password file <none>
I20250114 02:54:57.765146 2952 fs_manager.cc:362] Metadata directory not provided
I20250114 02:54:57.765365 2952 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 02:54:57.765818 2952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 02:54:57.770887 2952 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-0/data/instance:
uuid: "a8f3d9567c014741a7e628bab3e18e27"
format_stamp: "Formatted at 2025-01-14 02:54:57 on dist-test-slave-36vg"
I20250114 02:54:57.772015 2952 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-0/wal/instance:
uuid: "a8f3d9567c014741a7e628bab3e18e27"
format_stamp: "Formatted at 2025-01-14 02:54:57 on dist-test-slave-36vg"
I20250114 02:54:57.779182 2952 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20250114 02:54:57.784822 2968 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 02:54:57.785843 2952 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20250114 02:54:57.786145 2952 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-0/data,/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-0/wal
uuid: "a8f3d9567c014741a7e628bab3e18e27"
format_stamp: "Formatted at 2025-01-14 02:54:57 on dist-test-slave-36vg"
I20250114 02:54:57.786473 2952 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 02:54:57.853425 2952 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 02:54:57.854862 2952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 02:54:57.855310 2952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 02:54:57.923817 2952 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.126:40235
I20250114 02:54:57.923909 3019 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.126:40235 every 8 connection(s)
I20250114 02:54:57.926491 2952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-0/data/info.pb
I20250114 02:54:57.931404 3020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:57.934160 429 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrCUXNq/build/tsan/bin/kudu as pid 2952
I20250114 02:54:57.934552 429 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-0/wal/instance
I20250114 02:54:57.937016 429 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrCUXNq/build/tsan/bin/kudu
/tmp/dist-test-taskrCUXNq/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.107.125:38653
--webserver_interface=127.0.107.125
--webserver_port=0
--builtin_ntp_servers=127.0.107.84:38241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.107.126:40235,127.0.107.125:38653,127.0.107.124:44431
--hive_metastore_uris=thrift://127.0.0.1:33943
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250114 02:54:57.942247 3020 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } has no permanent_uuid. Determining permanent_uuid...
I20250114 02:54:57.967502 3020 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } has no permanent_uuid. Determining permanent_uuid...
W20250114 02:54:57.969846 2971 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)
W20250114 02:54:57.973749 3020 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.107.125:38653: Network error: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)
I20250114 02:54:58.023245 3020 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } attempt: 1
W20250114 02:54:58.027402 3020 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.107.125:38653: Network error: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)
I20250114 02:54:58.095839 3020 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } attempt: 2
W20250114 02:54:58.100296 3020 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.107.125:38653: Network error: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)
I20250114 02:54:58.191690 3020 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } attempt: 3
W20250114 02:54:58.195868 3020 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.107.125:38653: Network error: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)
W20250114 02:54:58.257519 3021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 02:54:58.258189 3021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 02:54:58.258639 3021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 02:54:58.289945 3021 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20250114 02:54:58.290200 3021 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20250114 02:54:58.290464 3021 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250114 02:54:58.290700 3021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 02:54:58.290964 3021 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250114 02:54:58.291211 3021 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250114 02:54:58.328188 3021 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.107.84:38241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:33943
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.107.126:40235,127.0.107.125:38653,127.0.107.124:44431
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.107.125:38653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.107.125
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type FASTDEBUG
built by None at 14 Jan 2025 02:43:23 UTC on a633016ff5c9
build id 4168
TSAN enabled
I20250114 02:54:58.329670 3021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 02:54:58.331256 3021 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250114 02:54:58.339319 3020 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } attempt: 4
W20250114 02:54:58.344945 3020 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.107.125:38653: Network error: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)
W20250114 02:54:58.347011 3031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 02:54:58.644486 3020 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } attempt: 5
W20250114 02:54:58.649972 3020 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.107.125:38653: Network error: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)
I20250114 02:54:59.197350 3020 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } attempt: 6
W20250114 02:54:59.209271 3020 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.107.125:38653: Network error: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)
W20250114 02:54:58.347743 3030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 02:54:58.348994 3033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 02:54:58.349669 3021 server_base.cc:1034] running on GCE node
I20250114 02:54:59.522799 3021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 02:54:59.525821 3021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 02:54:59.527278 3021 hybrid_clock.cc:648] HybridClock initialized: now 1736823299527249 us; error 45 us; skew 500 ppm
I20250114 02:54:59.528080 3021 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250114 02:54:59.534747 3021 webserver.cc:458] Webserver started at http://127.0.107.125:37807/ using document root <none> and password file <none>
I20250114 02:54:59.535637 3021 fs_manager.cc:362] Metadata directory not provided
I20250114 02:54:59.535837 3021 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 02:54:59.536293 3021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 02:54:59.541304 3021 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-1/data/instance:
uuid: "bfb2e2fecfb94af8bf892c6f774c2d89"
format_stamp: "Formatted at 2025-01-14 02:54:59 on dist-test-slave-36vg"
I20250114 02:54:59.542402 3021 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-1/wal/instance:
uuid: "bfb2e2fecfb94af8bf892c6f774c2d89"
format_stamp: "Formatted at 2025-01-14 02:54:59 on dist-test-slave-36vg"
I20250114 02:54:59.550029 3021 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20250114 02:54:59.555438 3042 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 02:54:59.556617 3021 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250114 02:54:59.556936 3021 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-1/data,/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-1/wal
uuid: "bfb2e2fecfb94af8bf892c6f774c2d89"
format_stamp: "Formatted at 2025-01-14 02:54:59 on dist-test-slave-36vg"
I20250114 02:54:59.557284 3021 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 02:54:59.620472 3021 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 02:54:59.621946 3021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 02:54:59.622383 3021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 02:54:59.689239 3021 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.125:38653
I20250114 02:54:59.689308 3093 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.125:38653 every 8 connection(s)
I20250114 02:54:59.691991 3021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-1/data/info.pb
I20250114 02:54:59.696964 3094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:54:59.697546 429 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrCUXNq/build/tsan/bin/kudu as pid 3021
I20250114 02:54:59.698055 429 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-1/wal/instance
I20250114 02:54:59.701684 429 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrCUXNq/build/tsan/bin/kudu
/tmp/dist-test-taskrCUXNq/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.107.124:44431
--webserver_interface=127.0.107.124
--webserver_port=0
--builtin_ntp_servers=127.0.107.84:38241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.107.126:40235,127.0.107.125:38653,127.0.107.124:44431
--hive_metastore_uris=thrift://127.0.0.1:33943
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250114 02:54:59.713079 3094 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } has no permanent_uuid. Determining permanent_uuid...
I20250114 02:54:59.734198 3094 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } has no permanent_uuid. Determining permanent_uuid...
I20250114 02:54:59.748984 3094 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } has no permanent_uuid. Determining permanent_uuid...
W20250114 02:54:59.751381 3046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.107.124:44431: connect: Connection refused (error 111)
W20250114 02:54:59.755221 3094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.107.124:44431: Network error: Client connection negotiation failed: client connection to 127.0.107.124:44431: connect: Connection refused (error 111)
I20250114 02:54:59.804730 3094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } attempt: 1
W20250114 02:54:59.808723 3094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.107.124:44431: Network error: Client connection negotiation failed: client connection to 127.0.107.124:44431: connect: Connection refused (error 111)
I20250114 02:54:59.877133 3094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } attempt: 2
W20250114 02:54:59.880944 3094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.107.124:44431: Network error: Client connection negotiation failed: client connection to 127.0.107.124:44431: connect: Connection refused (error 111)
I20250114 02:54:59.972450 3094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } attempt: 3
W20250114 02:54:59.976294 3094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.107.124:44431: Network error: Client connection negotiation failed: client connection to 127.0.107.124:44431: connect: Connection refused (error 111)
W20250114 02:55:00.013754 3095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 02:55:00.014312 3095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 02:55:00.014700 3095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 02:55:00.046146 3095 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20250114 02:55:00.046387 3095 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20250114 02:55:00.046589 3095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250114 02:55:00.046798 3095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 02:55:00.047000 3095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250114 02:55:00.047191 3095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250114 02:55:00.083709 3095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.107.84:38241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:33943
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.107.126:40235,127.0.107.125:38653,127.0.107.124:44431
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.107.124:44431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.107.124
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type FASTDEBUG
built by None at 14 Jan 2025 02:43:23 UTC on a633016ff5c9
build id 4168
TSAN enabled
I20250114 02:55:00.085009 3095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 02:55:00.086468 3095 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 02:55:00.103024 3105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 02:55:00.119807 3094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } attempt: 4
W20250114 02:55:00.123898 3094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.107.124:44431: Network error: Client connection negotiation failed: client connection to 127.0.107.124:44431: connect: Connection refused (error 111)
I20250114 02:55:00.269963 3020 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } attempt: 7
I20250114 02:55:00.282099 3020 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } has no permanent_uuid. Determining permanent_uuid...
W20250114 02:55:00.287346 3020 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.107.124:44431: Network error: Client connection negotiation failed: client connection to 127.0.107.124:44431: connect: Connection refused (error 111)
I20250114 02:55:00.346021 3020 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } attempt: 1
W20250114 02:55:00.350674 3020 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.107.124:44431: Network error: Client connection negotiation failed: client connection to 127.0.107.124:44431: connect: Connection refused (error 111)
I20250114 02:55:00.423529 3094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } attempt: 5
W20250114 02:55:00.428829 3094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.107.124:44431: Network error: Client connection negotiation failed: client connection to 127.0.107.124:44431: connect: Connection refused (error 111)
I20250114 02:55:00.432024 3020 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } attempt: 2
W20250114 02:55:00.436486 3020 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.107.124:44431: Network error: Client connection negotiation failed: client connection to 127.0.107.124:44431: connect: Connection refused (error 111)
I20250114 02:55:00.521869 3020 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } attempt: 3
W20250114 02:55:00.526250 3020 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.107.124:44431: Network error: Client connection negotiation failed: client connection to 127.0.107.124:44431: connect: Connection refused (error 111)
I20250114 02:55:00.666667 3020 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } attempt: 4
W20250114 02:55:00.671494 3020 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.107.124:44431: Network error: Client connection negotiation failed: client connection to 127.0.107.124:44431: connect: Connection refused (error 111)
I20250114 02:55:00.954983 3020 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } attempt: 5
W20250114 02:55:00.959259 3020 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.107.124:44431: Network error: Client connection negotiation failed: client connection to 127.0.107.124:44431: connect: Connection refused (error 111)
I20250114 02:55:00.976310 3094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } attempt: 6
W20250114 02:55:00.982769 3094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.107.124:44431: Network error: Client connection negotiation failed: client connection to 127.0.107.124:44431: connect: Connection refused (error 111)
W20250114 02:55:00.103864 3108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 02:55:00.105129 3106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 02:55:00.105872 3095 server_base.cc:1034] running on GCE node
I20250114 02:55:01.269995 3095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 02:55:01.272673 3095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 02:55:01.274012 3095 hybrid_clock.cc:648] HybridClock initialized: now 1736823301273993 us; error 50 us; skew 500 ppm
I20250114 02:55:01.274794 3095 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250114 02:55:01.281370 3095 webserver.cc:458] Webserver started at http://127.0.107.124:36967/ using document root <none> and password file <none>
I20250114 02:55:01.282266 3095 fs_manager.cc:362] Metadata directory not provided
I20250114 02:55:01.282474 3095 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 02:55:01.282892 3095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 02:55:01.288938 3095 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-2/data/instance:
uuid: "f9a6e2eab6e541eca12ee79c1d3004e8"
format_stamp: "Formatted at 2025-01-14 02:55:01 on dist-test-slave-36vg"
I20250114 02:55:01.290042 3095 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-2/wal/instance:
uuid: "f9a6e2eab6e541eca12ee79c1d3004e8"
format_stamp: "Formatted at 2025-01-14 02:55:01 on dist-test-slave-36vg"
I20250114 02:55:01.297005 3095 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20250114 02:55:01.302475 3120 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 02:55:01.303611 3095 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250114 02:55:01.303942 3095 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-2/data,/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-2/wal
uuid: "f9a6e2eab6e541eca12ee79c1d3004e8"
format_stamp: "Formatted at 2025-01-14 02:55:01 on dist-test-slave-36vg"
I20250114 02:55:01.304314 3095 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 02:55:01.379345 3095 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 02:55:01.380838 3095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 02:55:01.381266 3095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 02:55:01.450682 3095 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.124:44431
I20250114 02:55:01.450765 3171 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.124:44431 every 8 connection(s)
I20250114 02:55:01.453498 3095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-2/data/info.pb
I20250114 02:55:01.458539 3172 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:55:01.463409 429 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrCUXNq/build/tsan/bin/kudu as pid 3095
I20250114 02:55:01.463804 429 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-2/wal/instance
I20250114 02:55:01.473613 3172 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } has no permanent_uuid. Determining permanent_uuid...
I20250114 02:55:01.496714 3172 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } has no permanent_uuid. Determining permanent_uuid...
I20250114 02:55:01.505498 3172 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } has no permanent_uuid. Determining permanent_uuid...
I20250114 02:55:01.511638 3020 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } attempt: 6
I20250114 02:55:01.525102 3172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8: Bootstrap starting.
I20250114 02:55:01.533366 3172 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8: Neither blocks nor log segments found. Creating new log.
I20250114 02:55:01.535717 3172 log.cc:826] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8: Log is configured to *not* fsync() on all Append() calls
I20250114 02:55:01.536967 3020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27: Bootstrap starting.
I20250114 02:55:01.541325 3172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8: No bootstrap required, opened a new log
I20250114 02:55:01.544719 3020 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27: Neither blocks nor log segments found. Creating new log.
I20250114 02:55:01.547091 3020 log.cc:826] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27: Log is configured to *not* fsync() on all Append() calls
I20250114 02:55:01.551992 3020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27: No bootstrap required, opened a new log
I20250114 02:55:01.572813 3172 raft_consensus.cc:357] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:55:01.573750 3172 raft_consensus.cc:383] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:55:01.574064 3172 raft_consensus.cc:738] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9a6e2eab6e541eca12ee79c1d3004e8, State: Initialized, Role: FOLLOWER
I20250114 02:55:01.575024 3172 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:55:01.577782 3183 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } } }
I20250114 02:55:01.578434 3183 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [sys.catalog]: This master's current role is: FOLLOWER
I20250114 02:55:01.579922 3172 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [sys.catalog]: configured and running, proceeding with master startup.
I20250114 02:55:01.583468 3020 raft_consensus.cc:357] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:55:01.584460 3020 raft_consensus.cc:383] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:55:01.584817 3020 raft_consensus.cc:738] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8f3d9567c014741a7e628bab3e18e27, State: Initialized, Role: FOLLOWER
I20250114 02:55:01.585848 3020 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:55:01.591727 3188 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } } }
I20250114 02:55:01.592469 3188 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [sys.catalog]: This master's current role is: FOLLOWER
I20250114 02:55:01.593976 3020 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [sys.catalog]: configured and running, proceeding with master startup.
W20250114 02:55:01.615092 2974 tablet.cc:2367] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250114 02:55:01.636152 3126 tablet.cc:2367] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250114 02:55:01.637084 3203 catalog_manager.cc:1559] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250114 02:55:01.637986 3203 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250114 02:55:01.653666 3203 catalog_manager.cc:1559] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250114 02:55:01.654234 3203 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250114 02:55:01.663458 3183 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:55:01.664165 3183 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:55:01.618046 3095 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: OK, instance_id { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" instance_seqno: 1736823297904486 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250114 02:55:01.668066 3095 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250114 02:55:01.673522 2995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8f3d9567c014741a7e628bab3e18e27" is_pre_election: true
I20250114 02:55:01.674525 2995 raft_consensus.cc:2463] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9a6e2eab6e541eca12ee79c1d3004e8 in term 0.
I20250114 02:55:01.676519 3183 leader_election.cc:290] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235), bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653)
W20250114 02:55:01.677490 3203 catalog_manager.cc:1559] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250114 02:55:01.678124 3203 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250114 02:55:01.682268 3123 leader_election.cc:304] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8f3d9567c014741a7e628bab3e18e27, f9a6e2eab6e541eca12ee79c1d3004e8; no voters:
I20250114 02:55:01.685926 3183 raft_consensus.cc:2798] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:55:01.686597 3183 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:55:01.687597 3183 raft_consensus.cc:3054] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:55:01.701124 3183 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
W20250114 02:55:01.707791 3203 catalog_manager.cc:1559] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250114 02:55:01.708319 3203 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250114 02:55:01.710881 3183 leader_election.cc:290] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 1 election: Requested vote from peers a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235), bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653)
I20250114 02:55:01.711328 2995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8f3d9567c014741a7e628bab3e18e27"
I20250114 02:55:01.711889 2995 raft_consensus.cc:3054] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:55:01.721024 3215 catalog_manager.cc:796] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20250114 02:55:01.722893 3203 catalog_manager.cc:1559] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250114 02:55:01.723213 3203 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250114 02:55:01.723377 2995 raft_consensus.cc:2463] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9a6e2eab6e541eca12ee79c1d3004e8 in term 1.
W20250114 02:55:01.737277 3203 catalog_manager.cc:1559] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250114 02:55:01.737604 3203 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250114 02:55:01.751771 3203 catalog_manager.cc:1559] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250114 02:55:01.752107 3203 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250114 02:55:01.754969 3123 leader_election.cc:304] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8f3d9567c014741a7e628bab3e18e27, f9a6e2eab6e541eca12ee79c1d3004e8; no voters:
I20250114 02:55:01.756088 3183 raft_consensus.cc:2798] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 1 FOLLOWER]: Leader election won for term 1
W20250114 02:55:01.773401 3215 catalog_manager.cc:1559] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250114 02:55:01.773789 3215 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250114 02:55:01.778191 3183 raft_consensus.cc:695] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 1 LEADER]: Becoming Leader. State: Replica: f9a6e2eab6e541eca12ee79c1d3004e8, State: Running, Role: LEADER
I20250114 02:55:01.779184 3183 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
W20250114 02:55:01.787948 3215 catalog_manager.cc:1559] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250114 02:55:01.788303 3215 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250114 02:55:01.729481 2952 master_runner.cc:186] Error getting master registration for 127.0.107.125:38653: OK, instance_id { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" instance_seqno: 1736823299670605 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250114 02:55:01.795677 2952 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250114 02:55:01.797906 3203 catalog_manager.cc:1559] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250114 02:55:01.798235 3203 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250114 02:55:01.803879 3215 catalog_manager.cc:1559] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250114 02:55:01.804198 3215 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250114 02:55:01.806963 3217 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9a6e2eab6e541eca12ee79c1d3004e8. Latest consensus state: current_term: 1 leader_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } } }
I20250114 02:55:01.807741 3217 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [sys.catalog]: This master's current role is: LEADER
W20250114 02:55:01.812433 3203 catalog_manager.cc:1559] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250114 02:55:01.812731 3203 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250114 02:55:01.818400 3215 catalog_manager.cc:1559] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250114 02:55:01.818739 3215 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250114 02:55:01.836077 3215 catalog_manager.cc:1559] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250114 02:55:01.836416 3215 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250114 02:55:01.837318 3203 catalog_manager.cc:1559] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250114 02:55:01.837899 3203 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250114 02:55:01.843601 3219 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
W20250114 02:55:01.856341 3215 catalog_manager.cc:1559] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250114 02:55:01.856662 3215 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250114 02:55:01.861676 3219 catalog_manager.cc:1485] Initializing Kudu cluster ID...
W20250114 02:55:01.870904 3215 catalog_manager.cc:1559] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250114 02:55:01.871249 3215 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250114 02:55:01.889993 3215 catalog_manager.cc:1559] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250114 02:55:01.890331 3215 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250114 02:55:01.902701 2995 raft_consensus.cc:1270] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [term 1 FOLLOWER]: Refusing update from remote peer f9a6e2eab6e541eca12ee79c1d3004e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 02:55:01.904920 3217 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250114 02:55:01.904788 3215 catalog_manager.cc:1559] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250114 02:55:01.905644 3215 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250114 02:55:01.949944 3188 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9a6e2eab6e541eca12ee79c1d3004e8. Latest consensus state: current_term: 1 leader_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } } }
I20250114 02:55:01.959268 3188 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [sys.catalog]: This master's current role is: FOLLOWER
W20250114 02:55:01.955899 3215 catalog_manager.cc:1559] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250114 02:55:01.959888 3215 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250114 02:55:01.964583 3220 mvcc.cc:204] Tried to move back new op lower bound from 7114028244563947520 to 7114028244098293760. Current Snapshot: MvccSnapshot[applied={T|T < 7114028244563947520}]
W20250114 02:55:01.978988 3215 catalog_manager.cc:1559] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20250114 02:55:01.989562 3219 catalog_manager.cc:1348] Generated new cluster ID: 383d09ac0a7b47d1bf47222602c9cf34
I20250114 02:55:01.989903 3219 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250114 02:55:01.989665 3217 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } } }
I20250114 02:55:01.990422 3217 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [sys.catalog]: This master's current role is: LEADER
W20250114 02:55:01.990279 3215 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250114 02:55:01.991998 3188 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } } }
I20250114 02:55:01.992753 3188 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [sys.catalog]: This master's current role is: FOLLOWER
I20250114 02:55:02.019519 3215 catalog_manager.cc:1260] Loaded cluster ID: 383d09ac0a7b47d1bf47222602c9cf34
I20250114 02:55:02.020298 3215 catalog_manager.cc:1553] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27: loading cluster ID for follower catalog manager: success
W20250114 02:55:02.030787 3215 catalog_manager.cc:1577] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20250114 02:55:02.031577 3215 catalog_manager.cc:874] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20250114 02:55:02.043370 3094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } attempt: 7
I20250114 02:55:02.058506 3219 catalog_manager.cc:1371] Generated new certificate authority record
I20250114 02:55:02.061055 3219 catalog_manager.cc:1505] Loading token signing keys...
I20250114 02:55:02.076682 3215 catalog_manager.cc:1575] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27: acquiring CA information for follower catalog manager: success
W20250114 02:55:02.080930 3215 catalog_manager.cc:874] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20250114 02:55:02.098014 3094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89: Bootstrap starting.
I20250114 02:55:02.112521 3219 catalog_manager.cc:5899] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8: Generated new TSK 0
I20250114 02:55:02.113615 3219 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250114 02:55:02.118340 3219 catalog_manager.cc:1528] Loading latest processed Hive Metastore notification log event ID...
I20250114 02:55:02.122503 3219 catalog_manager.cc:5748] Last processed Hive Metastore notification event ID: -1
I20250114 02:55:02.124379 3094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89: Neither blocks nor log segments found. Creating new log.
I20250114 02:55:02.126762 3094 log.cc:826] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89: Log is configured to *not* fsync() on all Append() calls
I20250114 02:55:02.159132 429 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrCUXNq/build/tsan/bin/kudu
/tmp/dist-test-taskrCUXNq/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.107.65:0
--local_ip_for_outbound_sockets=127.0.107.65
--webserver_interface=127.0.107.65
--webserver_port=0
--tserver_master_addrs=127.0.107.126:40235,127.0.107.125:38653,127.0.107.124:44431
--builtin_ntp_servers=127.0.107.84:38241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250114 02:55:02.159417 3094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89: No bootstrap required, opened a new log
I20250114 02:55:02.167186 3215 catalog_manager.cc:1603] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250114 02:55:02.194705 3094 raft_consensus.cc:357] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:55:02.195809 3094 raft_consensus.cc:383] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:55:02.196190 3094 raft_consensus.cc:738] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfb2e2fecfb94af8bf892c6f774c2d89, State: Initialized, Role: FOLLOWER
I20250114 02:55:02.197243 3094 consensus_queue.cc:260] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:55:02.220017 3068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89"
I20250114 02:55:02.220816 3068 raft_consensus.cc:3054] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:55:02.223368 3069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" is_pre_election: true
W20250114 02:55:02.224483 3069 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44178 (request call id 1) took 540 ms (client timeout 679 ms). Trace:
W20250114 02:55:02.224818 3069 rpcz_store.cc:269] 0114 02:55:01.683915 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:55:01.685152 (+ 1237us) service_pool.cc:225] Handling call
0114 02:55:02.224408 (+539256us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:55:02.231717 3068 raft_consensus.cc:2463] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9a6e2eab6e541eca12ee79c1d3004e8 in term 1.
I20250114 02:55:02.233170 3067 raft_consensus.cc:1270] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [term 1 FOLLOWER]: Refusing update from remote peer f9a6e2eab6e541eca12ee79c1d3004e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250114 02:55:02.240456 3068 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44178 (request call id 2) took 523 ms (client timeout 595 ms). Trace:
I20250114 02:55:02.240700 3183 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250114 02:55:02.241961 3068 rpcz_store.cc:269] 0114 02:55:01.717298 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 02:55:01.717458 (+ 160us) service_pool.cc:225] Handling call
0114 02:55:02.240400 (+522942us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250114 02:55:02.240875 3229 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } } }
I20250114 02:55:02.242902 3229 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [sys.catalog]: This master's current role is: FOLLOWER
I20250114 02:55:02.241662 3094 sys_catalog.cc:564] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [sys.catalog]: configured and running, proceeding with master startup.
I20250114 02:55:02.313342 3229 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9a6e2eab6e541eca12ee79c1d3004e8. Latest consensus state: current_term: 1 leader_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } } }
I20250114 02:55:02.314143 3229 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [sys.catalog]: This master's current role is: FOLLOWER
I20250114 02:55:02.316165 3235 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } } }
I20250114 02:55:02.317081 3235 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [sys.catalog]: This master's current role is: FOLLOWER
I20250114 02:55:02.431010 3249 catalog_manager.cc:1260] Loaded cluster ID: 383d09ac0a7b47d1bf47222602c9cf34
I20250114 02:55:02.431389 3249 catalog_manager.cc:1553] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89: loading cluster ID for follower catalog manager: success
I20250114 02:55:02.440243 3249 catalog_manager.cc:1575] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89: acquiring CA information for follower catalog manager: success
I20250114 02:55:02.446959 3249 catalog_manager.cc:1603] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20250114 02:55:02.706751 3228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 02:55:02.707508 3228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 02:55:02.708557 3228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250114 02:55:02.750953 3235 raft_consensus.cc:491] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9a6e2eab6e541eca12ee79c1d3004e8)
I20250114 02:55:02.751448 3235 raft_consensus.cc:513] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:55:02.753805 3235 leader_election.cc:290] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235), f9a6e2eab6e541eca12ee79c1d3004e8 (127.0.107.124:44431)
I20250114 02:55:02.757120 2995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a8f3d9567c014741a7e628bab3e18e27" is_pre_election: true
I20250114 02:55:02.757706 3147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" is_pre_election: true
I20250114 02:55:02.759583 3046 leader_election.cc:304] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bfb2e2fecfb94af8bf892c6f774c2d89; no voters: a8f3d9567c014741a7e628bab3e18e27, f9a6e2eab6e541eca12ee79c1d3004e8
I20250114 02:55:02.760857 3235 raft_consensus.cc:2743] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250114 02:55:02.776293 3228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 02:55:02.785774 3228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.107.65
I20250114 02:55:02.810132 3183 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } } }
I20250114 02:55:02.810938 3183 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [sys.catalog]: This master's current role is: LEADER
I20250114 02:55:02.864512 3228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.107.84:38241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.107.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.107.65
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.107.126:40235,127.0.107.125:38653,127.0.107.124:44431
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.107.65
--log_dir=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type FASTDEBUG
built by None at 14 Jan 2025 02:43:23 UTC on a633016ff5c9
build id 4168
TSAN enabled
I20250114 02:55:02.866254 3228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 02:55:02.868368 3228 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 02:55:02.911285 3259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 02:55:03.084033 3263 raft_consensus.cc:491] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9a6e2eab6e541eca12ee79c1d3004e8)
I20250114 02:55:03.084775 3263 raft_consensus.cc:513] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:55:03.088042 3068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a8f3d9567c014741a7e628bab3e18e27" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" is_pre_election: true
I20250114 02:55:03.088779 3147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a8f3d9567c014741a7e628bab3e18e27" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" is_pre_election: true
I20250114 02:55:03.090356 2972 leader_election.cc:304] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a8f3d9567c014741a7e628bab3e18e27; no voters: bfb2e2fecfb94af8bf892c6f774c2d89, f9a6e2eab6e541eca12ee79c1d3004e8
I20250114 02:55:03.093463 3263 leader_election.cc:290] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653), f9a6e2eab6e541eca12ee79c1d3004e8 (127.0.107.124:44431)
I20250114 02:55:03.094419 3263 raft_consensus.cc:2743] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:55:03.657361 3265 raft_consensus.cc:491] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9a6e2eab6e541eca12ee79c1d3004e8)
I20250114 02:55:03.658270 3265 raft_consensus.cc:513] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:55:03.661183 3068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a8f3d9567c014741a7e628bab3e18e27" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" is_pre_election: true
I20250114 02:55:03.664167 3147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a8f3d9567c014741a7e628bab3e18e27" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" is_pre_election: true
I20250114 02:55:03.665731 2972 leader_election.cc:304] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a8f3d9567c014741a7e628bab3e18e27; no voters: bfb2e2fecfb94af8bf892c6f774c2d89, f9a6e2eab6e541eca12ee79c1d3004e8
I20250114 02:55:03.668303 3265 leader_election.cc:290] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653), f9a6e2eab6e541eca12ee79c1d3004e8 (127.0.107.124:44431)
I20250114 02:55:03.669158 3265 raft_consensus.cc:2743] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:55:04.273396 3267 raft_consensus.cc:491] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9a6e2eab6e541eca12ee79c1d3004e8)
I20250114 02:55:04.274091 3267 raft_consensus.cc:513] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:55:04.276772 3068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a8f3d9567c014741a7e628bab3e18e27" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" is_pre_election: true
I20250114 02:55:04.280625 3147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a8f3d9567c014741a7e628bab3e18e27" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" is_pre_election: true
I20250114 02:55:04.282195 2972 leader_election.cc:304] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a8f3d9567c014741a7e628bab3e18e27; no voters: bfb2e2fecfb94af8bf892c6f774c2d89, f9a6e2eab6e541eca12ee79c1d3004e8
I20250114 02:55:04.289166 3267 leader_election.cc:290] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653), f9a6e2eab6e541eca12ee79c1d3004e8 (127.0.107.124:44431)
I20250114 02:55:04.292444 3267 raft_consensus.cc:2743] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:55:04.537231 3269 raft_consensus.cc:491] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9a6e2eab6e541eca12ee79c1d3004e8)
I20250114 02:55:04.537945 3269 raft_consensus.cc:513] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:55:04.541568 2995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a8f3d9567c014741a7e628bab3e18e27" is_pre_election: true
I20250114 02:55:04.557339 3269 leader_election.cc:290] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235), f9a6e2eab6e541eca12ee79c1d3004e8 (127.0.107.124:44431)
I20250114 02:55:04.557859 3147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" is_pre_election: true
I20250114 02:55:04.559234 3046 leader_election.cc:304] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bfb2e2fecfb94af8bf892c6f774c2d89; no voters: a8f3d9567c014741a7e628bab3e18e27, f9a6e2eab6e541eca12ee79c1d3004e8
I20250114 02:55:04.561048 3269 raft_consensus.cc:2743] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250114 02:55:02.922780 3260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 02:55:04.293136 3258 debug-util.cc:398] Leaking SignalData structure 0x7b080001c400 after lost signal to thread 3228
W20250114 02:55:05.445463 3258 kernel_stack_watchdog.cc:198] Thread 3228 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:636 for 378ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250114 02:55:05.451347 3228 thread.cc:635] GCE (cloud detector) Time spent creating pthread: real 2.539s user 0.722s sys 1.359s
W20250114 02:55:05.452330 3228 thread.cc:601] GCE (cloud detector) Time spent starting thread: real 2.540s user 0.722s sys 1.359s
I20250114 02:55:05.463255 3228 server_base.cc:1034] running on GCE node
W20250114 02:55:05.468807 3272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 02:55:05.470316 3228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 02:55:05.490010 3228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 02:55:05.491612 3228 hybrid_clock.cc:648] HybridClock initialized: now 1736823305491530 us; error 85 us; skew 500 ppm
I20250114 02:55:05.492863 3228 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250114 02:55:05.492864 3273 raft_consensus.cc:491] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9a6e2eab6e541eca12ee79c1d3004e8)
I20250114 02:55:05.493351 3273 raft_consensus.cc:513] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:55:05.496285 2995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a8f3d9567c014741a7e628bab3e18e27" is_pre_election: true
I20250114 02:55:05.498584 3273 leader_election.cc:290] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235), f9a6e2eab6e541eca12ee79c1d3004e8 (127.0.107.124:44431)
I20250114 02:55:05.499136 3147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" is_pre_election: true
I20250114 02:55:05.500422 3046 leader_election.cc:304] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bfb2e2fecfb94af8bf892c6f774c2d89; no voters: a8f3d9567c014741a7e628bab3e18e27, f9a6e2eab6e541eca12ee79c1d3004e8
I20250114 02:55:05.506443 3273 raft_consensus.cc:2743] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:55:05.524904 3228 webserver.cc:458] Webserver started at http://127.0.107.65:35101/ using document root <none> and password file <none>
I20250114 02:55:05.526393 3228 fs_manager.cc:362] Metadata directory not provided
I20250114 02:55:05.526736 3228 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 02:55:05.527412 3228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 02:55:05.534875 3228 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-0/data/instance:
uuid: "7b7de0f9038c4ba287750ad132cb3597"
format_stamp: "Formatted at 2025-01-14 02:55:05 on dist-test-slave-36vg"
I20250114 02:55:05.536677 3228 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-0/wal/instance:
uuid: "7b7de0f9038c4ba287750ad132cb3597"
format_stamp: "Formatted at 2025-01-14 02:55:05 on dist-test-slave-36vg"
I20250114 02:55:05.566622 3228 fs_manager.cc:696] Time spent creating directory manager: real 0.029s user 0.012s sys 0.016s
I20250114 02:55:05.581346 3279 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 02:55:05.582787 3228 fs_manager.cc:730] Time spent opening block manager: real 0.011s user 0.004s sys 0.007s
I20250114 02:55:05.583199 3228 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-0/wal
uuid: "7b7de0f9038c4ba287750ad132cb3597"
format_stamp: "Formatted at 2025-01-14 02:55:05 on dist-test-slave-36vg"
I20250114 02:55:05.583632 3228 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 02:55:05.676156 3228 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 02:55:05.678134 3228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 02:55:05.678709 3228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 02:55:05.687889 3228 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 02:55:05.699806 3228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 02:55:05.700165 3228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 02:55:05.700592 3228 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 02:55:05.700800 3228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 02:55:06.022303 3228 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.65:34329
I20250114 02:55:06.022464 3391 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.65:34329 every 8 connection(s)
I20250114 02:55:06.025390 3228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-0/data/info.pb
I20250114 02:55:06.029685 429 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrCUXNq/build/tsan/bin/kudu as pid 3228
I20250114 02:55:06.030375 429 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-0/wal/instance
I20250114 02:55:06.060880 429 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrCUXNq/build/tsan/bin/kudu
/tmp/dist-test-taskrCUXNq/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.107.66:0
--local_ip_for_outbound_sockets=127.0.107.66
--webserver_interface=127.0.107.66
--webserver_port=0
--tserver_master_addrs=127.0.107.126:40235,127.0.107.125:38653,127.0.107.124:44431
--builtin_ntp_servers=127.0.107.84:38241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250114 02:55:06.107761 3398 heartbeater.cc:346] Connected to a master server at 127.0.107.126:40235
I20250114 02:55:06.108314 3398 heartbeater.cc:463] Registering TS with master...
I20250114 02:55:06.109591 3398 heartbeater.cc:510] Master 127.0.107.126:40235 requested a full tablet report, sending...
I20250114 02:55:06.113682 2985 ts_manager.cc:194] Registered new tserver with Master: 7b7de0f9038c4ba287750ad132cb3597 (127.0.107.65:34329)
I20250114 02:55:06.115401 3393 heartbeater.cc:346] Connected to a master server at 127.0.107.125:38653
I20250114 02:55:06.115768 3393 heartbeater.cc:463] Registering TS with master...
I20250114 02:55:06.116676 3393 heartbeater.cc:510] Master 127.0.107.125:38653 requested a full tablet report, sending...
I20250114 02:55:06.119459 3059 ts_manager.cc:194] Registered new tserver with Master: 7b7de0f9038c4ba287750ad132cb3597 (127.0.107.65:34329)
I20250114 02:55:06.122128 3392 heartbeater.cc:346] Connected to a master server at 127.0.107.124:44431
I20250114 02:55:06.122431 3392 heartbeater.cc:463] Registering TS with master...
I20250114 02:55:06.123306 3392 heartbeater.cc:510] Master 127.0.107.124:44431 requested a full tablet report, sending...
I20250114 02:55:06.125587 3137 ts_manager.cc:194] Registered new tserver with Master: 7b7de0f9038c4ba287750ad132cb3597 (127.0.107.65:34329)
I20250114 02:55:06.127943 3137 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.107.65:38355
W20250114 02:55:06.419605 3402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 02:55:06.420192 3402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 02:55:06.420821 3402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 02:55:06.453279 3402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 02:55:06.454205 3402 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.107.66
I20250114 02:55:06.493717 3402 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.107.84:38241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.107.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.107.66
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.107.126:40235,127.0.107.125:38653,127.0.107.124:44431
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.107.66
--log_dir=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type FASTDEBUG
built by None at 14 Jan 2025 02:43:23 UTC on a633016ff5c9
build id 4168
TSAN enabled
I20250114 02:55:06.495079 3402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 02:55:06.496704 3402 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 02:55:06.515329 3410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 02:55:06.610898 3415 raft_consensus.cc:491] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9a6e2eab6e541eca12ee79c1d3004e8)
I20250114 02:55:06.611582 3415 raft_consensus.cc:513] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:55:06.613664 3415 leader_election.cc:290] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653), f9a6e2eab6e541eca12ee79c1d3004e8 (127.0.107.124:44431)
I20250114 02:55:06.614482 3068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a8f3d9567c014741a7e628bab3e18e27" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" is_pre_election: true
I20250114 02:55:06.615135 3147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a8f3d9567c014741a7e628bab3e18e27" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" is_pre_election: true
I20250114 02:55:06.616771 2972 leader_election.cc:304] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a8f3d9567c014741a7e628bab3e18e27; no voters: bfb2e2fecfb94af8bf892c6f774c2d89, f9a6e2eab6e541eca12ee79c1d3004e8
I20250114 02:55:06.617434 3415 raft_consensus.cc:2743] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:55:06.631184 3392 heartbeater.cc:502] Master 127.0.107.124:44431 was elected leader, sending a full tablet report...
I20250114 02:55:07.229166 3416 raft_consensus.cc:491] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9a6e2eab6e541eca12ee79c1d3004e8)
I20250114 02:55:07.229532 3416 raft_consensus.cc:513] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:55:07.230980 3416 leader_election.cc:290] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653), f9a6e2eab6e541eca12ee79c1d3004e8 (127.0.107.124:44431)
I20250114 02:55:07.232038 3147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a8f3d9567c014741a7e628bab3e18e27" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" is_pre_election: true
I20250114 02:55:07.232285 3068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a8f3d9567c014741a7e628bab3e18e27" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" is_pre_election: true
I20250114 02:55:07.233613 2971 leader_election.cc:304] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a8f3d9567c014741a7e628bab3e18e27; no voters: bfb2e2fecfb94af8bf892c6f774c2d89, f9a6e2eab6e541eca12ee79c1d3004e8
I20250114 02:55:07.234232 3416 raft_consensus.cc:2743] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:55:07.371464 3417 raft_consensus.cc:491] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9a6e2eab6e541eca12ee79c1d3004e8)
I20250114 02:55:07.371852 3417 raft_consensus.cc:513] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:55:07.373468 3417 leader_election.cc:290] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235), f9a6e2eab6e541eca12ee79c1d3004e8 (127.0.107.124:44431)
I20250114 02:55:07.374413 2995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a8f3d9567c014741a7e628bab3e18e27" is_pre_election: true
I20250114 02:55:07.374903 3147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" is_pre_election: true
I20250114 02:55:07.376179 3046 leader_election.cc:304] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bfb2e2fecfb94af8bf892c6f774c2d89; no voters: a8f3d9567c014741a7e628bab3e18e27, f9a6e2eab6e541eca12ee79c1d3004e8
I20250114 02:55:07.376765 3417 raft_consensus.cc:2743] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250114 02:55:06.516129 3409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 02:55:06.518023 3402 server_base.cc:1034] running on GCE node
W20250114 02:55:06.517021 3412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 02:55:07.686908 3402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 02:55:07.689457 3402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 02:55:07.690936 3402 hybrid_clock.cc:648] HybridClock initialized: now 1736823307690858 us; error 88 us; skew 500 ppm
I20250114 02:55:07.692045 3402 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250114 02:55:07.699271 3402 webserver.cc:458] Webserver started at http://127.0.107.66:40469/ using document root <none> and password file <none>
I20250114 02:55:07.700587 3402 fs_manager.cc:362] Metadata directory not provided
I20250114 02:55:07.700899 3402 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 02:55:07.701436 3402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 02:55:07.708451 3402 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-1/data/instance:
uuid: "39c64942315c4716baaee0c251c05acd"
format_stamp: "Formatted at 2025-01-14 02:55:07 on dist-test-slave-36vg"
I20250114 02:55:07.709872 3402 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-1/wal/instance:
uuid: "39c64942315c4716baaee0c251c05acd"
format_stamp: "Formatted at 2025-01-14 02:55:07 on dist-test-slave-36vg"
I20250114 02:55:07.718734 3402 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.006s sys 0.004s
I20250114 02:55:07.725731 3422 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 02:55:07.726847 3402 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250114 02:55:07.727162 3402 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-1/wal
uuid: "39c64942315c4716baaee0c251c05acd"
format_stamp: "Formatted at 2025-01-14 02:55:07 on dist-test-slave-36vg"
I20250114 02:55:07.727491 3402 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 02:55:07.777630 3402 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 02:55:07.779189 3402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 02:55:07.779613 3402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 02:55:07.782373 3402 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 02:55:07.786533 3402 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 02:55:07.786788 3402 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 02:55:07.787043 3402 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 02:55:07.787217 3402 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 02:55:07.809233 3440 raft_consensus.cc:491] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9a6e2eab6e541eca12ee79c1d3004e8)
I20250114 02:55:07.809715 3440 raft_consensus.cc:513] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:55:07.811417 3440 leader_election.cc:290] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653), f9a6e2eab6e541eca12ee79c1d3004e8 (127.0.107.124:44431)
I20250114 02:55:07.812248 3068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a8f3d9567c014741a7e628bab3e18e27" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" is_pre_election: true
I20250114 02:55:07.812556 3147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a8f3d9567c014741a7e628bab3e18e27" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" is_pre_election: true
I20250114 02:55:07.813874 2972 leader_election.cc:304] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a8f3d9567c014741a7e628bab3e18e27; no voters: bfb2e2fecfb94af8bf892c6f774c2d89, f9a6e2eab6e541eca12ee79c1d3004e8
I20250114 02:55:07.814756 3440 raft_consensus.cc:2743] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:55:07.924789 3402 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.66:33487
I20250114 02:55:07.924924 3535 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.66:33487 every 8 connection(s)
I20250114 02:55:07.928449 3402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-1/data/info.pb
I20250114 02:55:07.930970 429 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrCUXNq/build/tsan/bin/kudu as pid 3402
I20250114 02:55:07.931509 429 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-1/wal/instance
I20250114 02:55:07.960986 429 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrCUXNq/build/tsan/bin/kudu
/tmp/dist-test-taskrCUXNq/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.107.67:0
--local_ip_for_outbound_sockets=127.0.107.67
--webserver_interface=127.0.107.67
--webserver_port=0
--tserver_master_addrs=127.0.107.126:40235,127.0.107.125:38653,127.0.107.124:44431
--builtin_ntp_servers=127.0.107.84:38241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250114 02:55:07.975368 3536 heartbeater.cc:346] Connected to a master server at 127.0.107.124:44431
I20250114 02:55:07.975831 3536 heartbeater.cc:463] Registering TS with master...
I20250114 02:55:07.977186 3536 heartbeater.cc:510] Master 127.0.107.124:44431 requested a full tablet report, sending...
I20250114 02:55:07.979974 3537 heartbeater.cc:346] Connected to a master server at 127.0.107.125:38653
I20250114 02:55:07.980335 3537 heartbeater.cc:463] Registering TS with master...
I20250114 02:55:07.980211 3137 ts_manager.cc:194] Registered new tserver with Master: 39c64942315c4716baaee0c251c05acd (127.0.107.66:33487)
I20250114 02:55:07.981159 3537 heartbeater.cc:510] Master 127.0.107.125:38653 requested a full tablet report, sending...
I20250114 02:55:07.981889 3137 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.107.66:54017
I20250114 02:55:07.983500 3059 ts_manager.cc:194] Registered new tserver with Master: 39c64942315c4716baaee0c251c05acd (127.0.107.66:33487)
I20250114 02:55:07.988991 3540 heartbeater.cc:346] Connected to a master server at 127.0.107.126:40235
I20250114 02:55:07.989334 3540 heartbeater.cc:463] Registering TS with master...
I20250114 02:55:07.989969 3540 heartbeater.cc:510] Master 127.0.107.126:40235 requested a full tablet report, sending...
I20250114 02:55:07.992674 2985 ts_manager.cc:194] Registered new tserver with Master: 39c64942315c4716baaee0c251c05acd (127.0.107.66:33487)
W20250114 02:55:08.294786 3546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 02:55:08.295315 3546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 02:55:08.295915 3546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 02:55:08.326241 3546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 02:55:08.327468 3546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.107.67
I20250114 02:55:08.361346 3546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.107.84:38241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.107.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.107.67
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.107.126:40235,127.0.107.125:38653,127.0.107.124:44431
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.107.67
--log_dir=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type FASTDEBUG
built by None at 14 Jan 2025 02:43:23 UTC on a633016ff5c9
build id 4168
TSAN enabled
I20250114 02:55:08.362879 3546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 02:55:08.364513 3546 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 02:55:08.381755 3553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 02:55:08.485698 3536 heartbeater.cc:502] Master 127.0.107.124:44431 was elected leader, sending a full tablet report...
I20250114 02:55:09.278676 3559 raft_consensus.cc:491] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9a6e2eab6e541eca12ee79c1d3004e8)
I20250114 02:55:09.279119 3559 raft_consensus.cc:513] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:55:09.280684 3559 leader_election.cc:290] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653), f9a6e2eab6e541eca12ee79c1d3004e8 (127.0.107.124:44431)
I20250114 02:55:09.281756 3147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a8f3d9567c014741a7e628bab3e18e27" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" is_pre_election: true
I20250114 02:55:09.282052 3068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a8f3d9567c014741a7e628bab3e18e27" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" is_pre_election: true
I20250114 02:55:09.283751 2971 leader_election.cc:304] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a8f3d9567c014741a7e628bab3e18e27; no voters: bfb2e2fecfb94af8bf892c6f774c2d89, f9a6e2eab6e541eca12ee79c1d3004e8
I20250114 02:55:09.284932 3559 raft_consensus.cc:2743] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 02:55:09.375284 3560 raft_consensus.cc:491] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9a6e2eab6e541eca12ee79c1d3004e8)
I20250114 02:55:09.375811 3560 raft_consensus.cc:513] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:55:09.377785 3560 leader_election.cc:290] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235), f9a6e2eab6e541eca12ee79c1d3004e8 (127.0.107.124:44431)
I20250114 02:55:09.379038 2995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a8f3d9567c014741a7e628bab3e18e27" is_pre_election: true
I20250114 02:55:09.379298 3147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" is_pre_election: true
I20250114 02:55:09.381399 3046 leader_election.cc:304] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bfb2e2fecfb94af8bf892c6f774c2d89; no voters: a8f3d9567c014741a7e628bab3e18e27, f9a6e2eab6e541eca12ee79c1d3004e8
I20250114 02:55:09.382274 3560 raft_consensus.cc:2743] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250114 02:55:08.382380 3556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 02:55:08.383355 3554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 02:55:08.384276 3546 server_base.cc:1034] running on GCE node
I20250114 02:55:09.581861 3546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 02:55:09.584101 3546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 02:55:09.585508 3546 hybrid_clock.cc:648] HybridClock initialized: now 1736823309585455 us; error 55 us; skew 500 ppm
I20250114 02:55:09.586301 3546 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250114 02:55:09.592667 3546 webserver.cc:458] Webserver started at http://127.0.107.67:35983/ using document root <none> and password file <none>
I20250114 02:55:09.593617 3546 fs_manager.cc:362] Metadata directory not provided
I20250114 02:55:09.593847 3546 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 02:55:09.594305 3546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 02:55:09.599406 3546 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-2/data/instance:
uuid: "5017c179ddc640fc9ebf5dea7f3957f8"
format_stamp: "Formatted at 2025-01-14 02:55:09 on dist-test-slave-36vg"
I20250114 02:55:09.600569 3546 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-2/wal/instance:
uuid: "5017c179ddc640fc9ebf5dea7f3957f8"
format_stamp: "Formatted at 2025-01-14 02:55:09 on dist-test-slave-36vg"
I20250114 02:55:09.608197 3546 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20250114 02:55:09.613822 3565 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 02:55:09.614907 3546 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250114 02:55:09.615245 3546 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-2/wal
uuid: "5017c179ddc640fc9ebf5dea7f3957f8"
format_stamp: "Formatted at 2025-01-14 02:55:09 on dist-test-slave-36vg"
I20250114 02:55:09.615598 3546 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 02:55:09.687337 3546 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 02:55:09.689175 3546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 02:55:09.689733 3546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 02:55:09.693090 3546 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 02:55:09.698748 3546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 02:55:09.699096 3546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 02:55:09.699457 3546 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 02:55:09.699692 3546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 02:55:09.852617 3546 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.67:45409
I20250114 02:55:09.852727 3677 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.67:45409 every 8 connection(s)
I20250114 02:55:09.855052 3546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-2/data/info.pb
I20250114 02:55:09.861845 429 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrCUXNq/build/tsan/bin/kudu as pid 3546
I20250114 02:55:09.862496 429 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-2/wal/instance
I20250114 02:55:09.895319 3679 heartbeater.cc:346] Connected to a master server at 127.0.107.125:38653
I20250114 02:55:09.895821 3679 heartbeater.cc:463] Registering TS with master...
I20250114 02:55:09.897221 3679 heartbeater.cc:510] Master 127.0.107.125:38653 requested a full tablet report, sending...
I20250114 02:55:09.898129 3678 heartbeater.cc:346] Connected to a master server at 127.0.107.124:44431
I20250114 02:55:09.898478 3678 heartbeater.cc:463] Registering TS with master...
I20250114 02:55:09.899338 3678 heartbeater.cc:510] Master 127.0.107.124:44431 requested a full tablet report, sending...
I20250114 02:55:09.900233 3059 ts_manager.cc:194] Registered new tserver with Master: 5017c179ddc640fc9ebf5dea7f3957f8 (127.0.107.67:45409)
I20250114 02:55:09.901561 3136 ts_manager.cc:194] Registered new tserver with Master: 5017c179ddc640fc9ebf5dea7f3957f8 (127.0.107.67:45409)
I20250114 02:55:09.902958 3680 heartbeater.cc:346] Connected to a master server at 127.0.107.126:40235
I20250114 02:55:09.903303 3680 heartbeater.cc:463] Registering TS with master...
I20250114 02:55:09.903244 3136 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.107.67:42009
I20250114 02:55:09.904134 3680 heartbeater.cc:510] Master 127.0.107.126:40235 requested a full tablet report, sending...
I20250114 02:55:09.906673 2984 ts_manager.cc:194] Registered new tserver with Master: 5017c179ddc640fc9ebf5dea7f3957f8 (127.0.107.67:45409)
I20250114 02:55:09.924974 429 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250114 02:55:09.978053 3136 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60016:
name: "default.Table_8aa1373bd9824b1a8cbed27ede045e55"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:55:09.981837 3135 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60016:
name: "default.Table_bab0c0670ef34820b254580eeb9eef70"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:55:09.983058 3137 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60016:
name: "default.Table_627b0faab8eb44c58814768e6dd28c15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:55:09.983281 3134 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60016:
name: "default.Table_b4cf0d29abe14186b59a52f5bcf7d11d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:55:09.993860 3136 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_8aa1373bd9824b1a8cbed27ede045e55 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:55:09.995570 3708 raft_consensus.cc:491] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9a6e2eab6e541eca12ee79c1d3004e8)
I20250114 02:55:09.996134 3708 raft_consensus.cc:513] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:55:10.000311 3708 leader_election.cc:290] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235), f9a6e2eab6e541eca12ee79c1d3004e8 (127.0.107.124:44431)
I20250114 02:55:10.000977 2995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a8f3d9567c014741a7e628bab3e18e27" is_pre_election: true
I20250114 02:55:10.001823 3147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" is_pre_election: true
I20250114 02:55:10.003660 3046 leader_election.cc:304] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bfb2e2fecfb94af8bf892c6f774c2d89; no voters: a8f3d9567c014741a7e628bab3e18e27, f9a6e2eab6e541eca12ee79c1d3004e8
W20250114 02:55:10.005647 3135 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_bab0c0670ef34820b254580eeb9eef70 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250114 02:55:10.007488 3134 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_b4cf0d29abe14186b59a52f5bcf7d11d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:55:10.008316 3708 raft_consensus.cc:2743] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250114 02:55:10.009233 3137 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_627b0faab8eb44c58814768e6dd28c15 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:55:10.054996 429 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrCUXNq/build/tsan/bin/kudu with pid 3095
W20250114 02:55:10.091224 3691 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.107.124:44431)
W20250114 02:55:10.139760 3691 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.CreateTable -> {remote=127.0.107.124:44431, user_credentials={real_user=slave}} blocked reactor thread for 50552.1us
W20250114 02:55:10.158432 3392 heartbeater.cc:643] Failed to heartbeat to 127.0.107.124:44431 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.107.124:44431: connect: Connection refused (error 111)
I20250114 02:55:10.305409 3715 raft_consensus.cc:491] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9a6e2eab6e541eca12ee79c1d3004e8)
I20250114 02:55:10.306135 3715 raft_consensus.cc:513] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:55:10.308836 3068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a8f3d9567c014741a7e628bab3e18e27" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" is_pre_election: true
I20250114 02:55:10.309453 3068 raft_consensus.cc:2463] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8f3d9567c014741a7e628bab3e18e27 in term 1.
I20250114 02:55:10.309707 3715 leader_election.cc:290] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653), f9a6e2eab6e541eca12ee79c1d3004e8 (127.0.107.124:44431)
I20250114 02:55:10.310776 2971 leader_election.cc:304] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8f3d9567c014741a7e628bab3e18e27, bfb2e2fecfb94af8bf892c6f774c2d89; no voters:
I20250114 02:55:10.311604 3715 raft_consensus.cc:2798] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 02:55:10.311975 3715 raft_consensus.cc:491] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f9a6e2eab6e541eca12ee79c1d3004e8)
I20250114 02:55:10.312422 3715 raft_consensus.cc:3054] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [term 1 FOLLOWER]: Advancing to term 2
W20250114 02:55:10.322085 2972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.107.124:44431: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20250114 02:55:10.326673 2972 leader_election.cc:336] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f9a6e2eab6e541eca12ee79c1d3004e8 (127.0.107.124:44431): Network error: Client connection negotiation failed: client connection to 127.0.107.124:44431: connect: Connection refused (error 111)
I20250114 02:55:10.326234 3715 raft_consensus.cc:513] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:55:10.333346 3068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a8f3d9567c014741a7e628bab3e18e27" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89"
I20250114 02:55:10.333636 3715 leader_election.cc:290] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [CANDIDATE]: Term 2 election: Requested vote from peers bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653), f9a6e2eab6e541eca12ee79c1d3004e8 (127.0.107.124:44431)
I20250114 02:55:10.333904 3068 raft_consensus.cc:3054] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [term 1 FOLLOWER]: Advancing to term 2
W20250114 02:55:10.336536 2972 leader_election.cc:336] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f9a6e2eab6e541eca12ee79c1d3004e8 (127.0.107.124:44431): Network error: Client connection negotiation failed: client connection to 127.0.107.124:44431: connect: Connection refused (error 111)
I20250114 02:55:10.342474 3068 raft_consensus.cc:2463] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8f3d9567c014741a7e628bab3e18e27 in term 2.
I20250114 02:55:10.343822 2971 leader_election.cc:304] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a8f3d9567c014741a7e628bab3e18e27, bfb2e2fecfb94af8bf892c6f774c2d89; no voters: f9a6e2eab6e541eca12ee79c1d3004e8
I20250114 02:55:10.345085 3715 raft_consensus.cc:2798] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 02:55:10.356401 3715 raft_consensus.cc:695] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [term 2 LEADER]: Becoming Leader. State: Replica: a8f3d9567c014741a7e628bab3e18e27, State: Running, Role: LEADER
I20250114 02:55:10.357517 3715 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:55:10.364470 3718 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8f3d9567c014741a7e628bab3e18e27. Latest consensus state: current_term: 2 leader_uuid: "a8f3d9567c014741a7e628bab3e18e27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } } }
I20250114 02:55:10.365258 3718 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [sys.catalog]: This master's current role is: LEADER
I20250114 02:55:10.369100 3720 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250114 02:55:10.377197 3720 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250114 02:55:10.381083 3720 catalog_manager.cc:1260] Loaded cluster ID: 383d09ac0a7b47d1bf47222602c9cf34
I20250114 02:55:10.381421 3720 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250114 02:55:10.386188 3720 catalog_manager.cc:1505] Loading token signing keys...
I20250114 02:55:10.391604 3720 catalog_manager.cc:5910] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27: Loaded TSK: 0
I20250114 02:55:10.393112 3720 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250114 02:55:10.396926 3720 catalog_manager.cc:1528] Loading latest processed Hive Metastore notification log event ID...
I20250114 02:55:10.400697 3720 catalog_manager.cc:5748] Last processed Hive Metastore notification event ID: -1
I20250114 02:55:10.406621 3678 heartbeater.cc:502] Master 127.0.107.124:44431 was elected leader, sending a full tablet report...
W20250114 02:55:10.410566 3678 heartbeater.cc:643] Failed to heartbeat to 127.0.107.124:44431 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.107.124:44431: connect: Connection refused (error 111)
I20250114 02:55:10.420984 2985 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57960:
name: "default.Table_bab0c0670ef34820b254580eeb9eef70"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:55:10.422845 2983 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57960:
name: "default.Table_627b0faab8eb44c58814768e6dd28c15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:55:10.422962 2982 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57960:
name: "default.Table_8aa1373bd9824b1a8cbed27ede045e55"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:55:10.420984 2984 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57960:
name: "default.Table_b4cf0d29abe14186b59a52f5bcf7d11d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 02:55:10.433703 2985 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_bab0c0670ef34820b254580eeb9eef70 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250114 02:55:10.507798 3536 heartbeater.cc:643] Failed to heartbeat to 127.0.107.124:44431 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.107.124:44431: connect: Connection refused (error 111)
I20250114 02:55:10.556334 2985 catalog_manager.cc:2149] created HMS catalog entry for table [id=89d59ce4ef604cce9b1e9a01a4aea791]
I20250114 02:55:10.564508 3068 raft_consensus.cc:1270] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [term 2 FOLLOWER]: Refusing update from remote peer a8f3d9567c014741a7e628bab3e18e27: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
W20250114 02:55:10.565356 2972 consensus_peers.cc:487] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 -> Peer f9a6e2eab6e541eca12ee79c1d3004e8 (127.0.107.124:44431): Couldn't send request to peer f9a6e2eab6e541eca12ee79c1d3004e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.124:44431: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250114 02:55:10.566952 3715 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20250114 02:55:10.568368 2982 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_8aa1373bd9824b1a8cbed27ede045e55 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250114 02:55:10.569187 2983 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_627b0faab8eb44c58814768e6dd28c15 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250114 02:55:10.573272 2984 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_b4cf0d29abe14186b59a52f5bcf7d11d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 02:55:10.596365 3724 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8f3d9567c014741a7e628bab3e18e27. Latest consensus state: current_term: 2 leader_uuid: "a8f3d9567c014741a7e628bab3e18e27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } } }
I20250114 02:55:10.597280 3724 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [sys.catalog]: This master's current role is: FOLLOWER
I20250114 02:55:10.602495 3718 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "a8f3d9567c014741a7e628bab3e18e27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } } }
I20250114 02:55:10.603240 3718 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27 [sys.catalog]: This master's current role is: LEADER
I20250114 02:55:10.605912 3724 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "a8f3d9567c014741a7e628bab3e18e27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } } }
I20250114 02:55:10.606693 3724 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [sys.catalog]: This master's current role is: FOLLOWER
I20250114 02:55:10.613780 2985 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57960:
table { table_name: "default.Table_bab0c0670ef34820b254580eeb9eef70" } new_table_name: "default.Table_91d61e17dfec48479865a7feb0460bd9" modify_external_catalogs: true
I20250114 02:55:10.650794 3327 tablet_service.cc:1467] Processing CreateTablet for tablet 55ce9a7d0c0f45bf982b5c064ee1f676 (DEFAULT_TABLE table=default.table_bab0c0670ef34820b254580eeb9eef70 [id=89d59ce4ef604cce9b1e9a01a4aea791]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:55:10.656391 3327 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55ce9a7d0c0f45bf982b5c064ee1f676. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:55:10.662914 3471 tablet_service.cc:1467] Processing CreateTablet for tablet 55ce9a7d0c0f45bf982b5c064ee1f676 (DEFAULT_TABLE table=default.table_bab0c0670ef34820b254580eeb9eef70 [id=89d59ce4ef604cce9b1e9a01a4aea791]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:55:10.664918 3471 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55ce9a7d0c0f45bf982b5c064ee1f676. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:55:10.676810 3613 tablet_service.cc:1467] Processing CreateTablet for tablet 55ce9a7d0c0f45bf982b5c064ee1f676 (DEFAULT_TABLE table=default.table_bab0c0670ef34820b254580eeb9eef70 [id=89d59ce4ef604cce9b1e9a01a4aea791]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:55:10.678701 3613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55ce9a7d0c0f45bf982b5c064ee1f676. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:55:10.699054 3736 tablet_bootstrap.cc:492] T 55ce9a7d0c0f45bf982b5c064ee1f676 P 39c64942315c4716baaee0c251c05acd: Bootstrap starting.
I20250114 02:55:10.708406 3736 tablet_bootstrap.cc:654] T 55ce9a7d0c0f45bf982b5c064ee1f676 P 39c64942315c4716baaee0c251c05acd: Neither blocks nor log segments found. Creating new log.
I20250114 02:55:10.710922 3736 log.cc:826] T 55ce9a7d0c0f45bf982b5c064ee1f676 P 39c64942315c4716baaee0c251c05acd: Log is configured to *not* fsync() on all Append() calls
I20250114 02:55:10.717283 3736 tablet_bootstrap.cc:492] T 55ce9a7d0c0f45bf982b5c064ee1f676 P 39c64942315c4716baaee0c251c05acd: No bootstrap required, opened a new log
I20250114 02:55:10.717931 3736 ts_tablet_manager.cc:1397] T 55ce9a7d0c0f45bf982b5c064ee1f676 P 39c64942315c4716baaee0c251c05acd: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.005s
I20250114 02:55:10.735287 2981 master_service.cc:946] ReplaceTablet: received request to replace tablet 55ce9a7d0c0f45bf982b5c064ee1f676 from {username='slave'} at 127.0.0.1:57960
I20250114 02:55:10.736788 3738 tablet_bootstrap.cc:492] T 55ce9a7d0c0f45bf982b5c064ee1f676 P 7b7de0f9038c4ba287750ad132cb3597: Bootstrap starting.
I20250114 02:55:10.740113 2980 master_service.cc:946] ReplaceTablet: received request to replace tablet 55ce9a7d0c0f45bf982b5c064ee1f676 from {username='slave'} at 127.0.0.1:57960
I20250114 02:55:10.745164 3738 tablet_bootstrap.cc:654] T 55ce9a7d0c0f45bf982b5c064ee1f676 P 7b7de0f9038c4ba287750ad132cb3597: Neither blocks nor log segments found. Creating new log.
I20250114 02:55:10.747495 3738 log.cc:826] T 55ce9a7d0c0f45bf982b5c064ee1f676 P 7b7de0f9038c4ba287750ad132cb3597: Log is configured to *not* fsync() on all Append() calls
I20250114 02:55:10.752101 3739 tablet_bootstrap.cc:492] T 55ce9a7d0c0f45bf982b5c064ee1f676 P 5017c179ddc640fc9ebf5dea7f3957f8: Bootstrap starting.
I20250114 02:55:10.765862 3739 tablet_bootstrap.cc:654] T 55ce9a7d0c0f45bf982b5c064ee1f676 P 5017c179ddc640fc9ebf5dea7f3957f8: Neither blocks nor log segments found. Creating new log.
I20250114 02:55:10.771386 3739 log.cc:826] T 55ce9a7d0c0f45bf982b5c064ee1f676 P 5017c179ddc640fc9ebf5dea7f3957f8: Log is configured to *not* fsync() on all Append() calls
I20250114 02:55:10.775314 2981 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27: Sending DeleteTablet for 3 replicas of tablet 55ce9a7d0c0f45bf982b5c064ee1f676
I20250114 02:55:10.775381 3736 raft_consensus.cc:357] T 55ce9a7d0c0f45bf982b5c064ee1f676 P 39c64942315c4716baaee0c251c05acd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } } peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } }
I20250114 02:55:10.776427 3736 raft_consensus.cc:383] T 55ce9a7d0c0f45bf982b5c064ee1f676 P 39c64942315c4716baaee0c251c05acd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:55:10.776770 3736 raft_consensus.cc:738] T 55ce9a7d0c0f45bf982b5c064ee1f676 P 39c64942315c4716baaee0c251c05acd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39c64942315c4716baaee0c251c05acd, State: Initialized, Role: FOLLOWER
I20250114 02:55:10.778416 3613 tablet_service.cc:1514] Processing DeleteTablet for tablet 55ce9a7d0c0f45bf982b5c064ee1f676 with delete_type TABLET_DATA_DELETED (replaced by tablet c174bcbc5d51402b8ae332b87ffa2139) from {username='slave'} at 127.0.0.1:60674
I20250114 02:55:10.780539 3327 tablet_service.cc:1514] Processing DeleteTablet for tablet 55ce9a7d0c0f45bf982b5c064ee1f676 with delete_type TABLET_DATA_DELETED (replaced by tablet c174bcbc5d51402b8ae332b87ffa2139) from {username='slave'} at 127.0.0.1:58826
I20250114 02:55:10.777819 3471 tablet_service.cc:1514] Processing DeleteTablet for tablet 55ce9a7d0c0f45bf982b5c064ee1f676 with delete_type TABLET_DATA_DELETED (replaced by tablet c174bcbc5d51402b8ae332b87ffa2139) from {username='slave'} at 127.0.0.1:58610
W20250114 02:55:10.788700 2969 catalog_manager.cc:4852] TS 5017c179ddc640fc9ebf5dea7f3957f8 (127.0.107.67:45409): delete failed for tablet 55ce9a7d0c0f45bf982b5c064ee1f676 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 55ce9a7d0c0f45bf982b5c064ee1f676 already in progress: creating tablet
I20250114 02:55:10.791134 3736 consensus_queue.cc:260] T 55ce9a7d0c0f45bf982b5c064ee1f676 P 39c64942315c4716baaee0c251c05acd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } } peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } }
I20250114 02:55:10.798461 3738 tablet_bootstrap.cc:492] T 55ce9a7d0c0f45bf982b5c064ee1f676 P 7b7de0f9038c4ba287750ad132cb3597: No bootstrap required, opened a new log
I20250114 02:55:10.799120 3738 ts_tablet_manager.cc:1397] T 55ce9a7d0c0f45bf982b5c064ee1f676 P 7b7de0f9038c4ba287750ad132cb3597: Time spent bootstrapping tablet: real 0.063s user 0.010s sys 0.005s
I20250114 02:55:10.808734 2980 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57960:
name: "default.Table_6e897f928c824eedbe6f75b0e4ad6b93"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:55:10.810070 2981 catalog_manager.cc:6596] ReplaceTablet: tablet 55ce9a7d0c0f45bf982b5c064ee1f676 deleted and replaced by tablet c174bcbc5d51402b8ae332b87ffa2139
I20250114 02:55:10.810168 3470 tablet_service.cc:1467] Processing CreateTablet for tablet c174bcbc5d51402b8ae332b87ffa2139 (DEFAULT_TABLE table=default.table_bab0c0670ef34820b254580eeb9eef70 [id=89d59ce4ef604cce9b1e9a01a4aea791]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:55:10.811331 3470 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c174bcbc5d51402b8ae332b87ffa2139. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:55:10.813966 3613 tablet_service.cc:1467] Processing CreateTablet for tablet c174bcbc5d51402b8ae332b87ffa2139 (DEFAULT_TABLE table=default.table_bab0c0670ef34820b254580eeb9eef70 [id=89d59ce4ef604cce9b1e9a01a4aea791]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:55:10.815002 3613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c174bcbc5d51402b8ae332b87ffa2139. 1 dirs total, 0 dirs full, 0 dirs failed
W20250114 02:55:10.822604 2970 catalog_manager.cc:4852] TS 39c64942315c4716baaee0c251c05acd (127.0.107.66:33487): delete failed for tablet 55ce9a7d0c0f45bf982b5c064ee1f676 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 55ce9a7d0c0f45bf982b5c064ee1f676 already in progress: creating tablet
I20250114 02:55:10.836633 3326 tablet_service.cc:1467] Processing CreateTablet for tablet c174bcbc5d51402b8ae332b87ffa2139 (DEFAULT_TABLE table=default.table_bab0c0670ef34820b254580eeb9eef70 [id=89d59ce4ef604cce9b1e9a01a4aea791]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:55:10.837905 3326 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c174bcbc5d51402b8ae332b87ffa2139. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:55:10.842434 3540 heartbeater.cc:502] Master 127.0.107.126:40235 was elected leader, sending a full tablet report...
I20250114 02:55:10.847568 3739 tablet_bootstrap.cc:492] T 55ce9a7d0c0f45bf982b5c064ee1f676 P 5017c179ddc640fc9ebf5dea7f3957f8: No bootstrap required, opened a new log
I20250114 02:55:10.848161 3739 ts_tablet_manager.cc:1397] T 55ce9a7d0c0f45bf982b5c064ee1f676 P 5017c179ddc640fc9ebf5dea7f3957f8: Time spent bootstrapping tablet: real 0.099s user 0.042s sys 0.012s
I20250114 02:55:10.852142 3470 tablet_service.cc:1514] Processing DeleteTablet for tablet 55ce9a7d0c0f45bf982b5c064ee1f676 with delete_type TABLET_DATA_DELETED (replaced by tablet c174bcbc5d51402b8ae332b87ffa2139) from {username='slave'} at 127.0.0.1:58610
W20250114 02:55:10.853852 2970 catalog_manager.cc:4852] TS 39c64942315c4716baaee0c251c05acd (127.0.107.66:33487): delete failed for tablet 55ce9a7d0c0f45bf982b5c064ee1f676 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 55ce9a7d0c0f45bf982b5c064ee1f676 already in progress: creating tablet
I20250114 02:55:10.857406 3736 ts_tablet_manager.cc:1428] T 55ce9a7d0c0f45bf982b5c064ee1f676 P 39c64942315c4716baaee0c251c05acd: Time spent starting tablet: real 0.139s user 0.029s sys 0.012s
W20250114 02:55:10.860798 2969 catalog_manager.cc:4852] TS 7b7de0f9038c4ba287750ad132cb3597 (127.0.107.65:34329): delete failed for tablet 55ce9a7d0c0f45bf982b5c064ee1f676 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 55ce9a7d0c0f45bf982b5c064ee1f676 already in progress: creating tablet
I20250114 02:55:10.873966 3470 tablet_service.cc:1514] Processing DeleteTablet for tablet 55ce9a7d0c0f45bf982b5c064ee1f676 with delete_type TABLET_DATA_DELETED (replaced by tablet c174bcbc5d51402b8ae332b87ffa2139) from {username='slave'} at 127.0.0.1:58610
I20250114 02:55:10.874928 3743 tablet_replica.cc:331] T 55ce9a7d0c0f45bf982b5c064ee1f676 P 39c64942315c4716baaee0c251c05acd: stopping tablet replica
I20250114 02:55:10.875748 3743 raft_consensus.cc:2238] T 55ce9a7d0c0f45bf982b5c064ee1f676 P 39c64942315c4716baaee0c251c05acd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250114 02:55:10.876372 3743 raft_consensus.cc:2267] T 55ce9a7d0c0f45bf982b5c064ee1f676 P 39c64942315c4716baaee0c251c05acd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250114 02:55:10.879073 3743 ts_tablet_manager.cc:1905] T 55ce9a7d0c0f45bf982b5c064ee1f676 P 39c64942315c4716baaee0c251c05acd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:55:10.878166 3736 tablet_bootstrap.cc:492] T c174bcbc5d51402b8ae332b87ffa2139 P 39c64942315c4716baaee0c251c05acd: Bootstrap starting.
02:55:10.885 [HMS - ERROR - pool-6-thread-2] (RetryingHMSHandler.java:201) AlreadyExistsException(message:Table hive.default.table_627b0faab8eb44c58814768e6dd28c15 already exists)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1839)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1786)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:2035)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy25.create_table_with_environment_context(Unknown Source)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14800)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14784)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:104)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
I20250114 02:55:10.890228 3736 tablet_bootstrap.cc:654] T c174bcbc5d51402b8ae332b87ffa2139 P 39c64942315c4716baaee0c251c05acd: Neither blocks nor log segments found. Creating new log.
W20250114 02:55:10.891350 2983 catalog_manager.cc:2145] Already present: failed to create HMS catalog entry for table [id=a44b7786f6b24861bb94bc574c58e595]: failed to create Hive MetaStore table: TException - service has thrown: AlreadyExistsException(message=Table hive.default.table_627b0faab8eb44c58814768e6dd28c15 already exists)
I20250114 02:55:10.896231 2983 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57960:
name: "default.Table_1e2047182d2241f4b619010ca10e11fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:55:10.901819 3736 tablet_bootstrap.cc:492] T c174bcbc5d51402b8ae332b87ffa2139 P 39c64942315c4716baaee0c251c05acd: No bootstrap required, opened a new log
I20250114 02:55:10.902343 3736 ts_tablet_manager.cc:1397] T c174bcbc5d51402b8ae332b87ffa2139 P 39c64942315c4716baaee0c251c05acd: Time spent bootstrapping tablet: real 0.024s user 0.011s sys 0.005s
I20250114 02:55:10.904400 3743 ts_tablet_manager.cc:1918] T 55ce9a7d0c0f45bf982b5c064ee1f676 P 39c64942315c4716baaee0c251c05acd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:55:10.905092 3736 raft_consensus.cc:357] T c174bcbc5d51402b8ae332b87ffa2139 P 39c64942315c4716baaee0c251c05acd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:10.906059 3736 raft_consensus.cc:383] T c174bcbc5d51402b8ae332b87ffa2139 P 39c64942315c4716baaee0c251c05acd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:55:10.906453 3736 raft_consensus.cc:738] T c174bcbc5d51402b8ae332b87ffa2139 P 39c64942315c4716baaee0c251c05acd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39c64942315c4716baaee0c251c05acd, State: Initialized, Role: FOLLOWER
I20250114 02:55:10.905669 3743 log.cc:1198] T 55ce9a7d0c0f45bf982b5c064ee1f676 P 39c64942315c4716baaee0c251c05acd: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/55ce9a7d0c0f45bf982b5c064ee1f676
I20250114 02:55:10.907833 3743 ts_tablet_manager.cc:1939] T 55ce9a7d0c0f45bf982b5c064ee1f676 P 39c64942315c4716baaee0c251c05acd: Deleting consensus metadata
I20250114 02:55:10.912997 3680 heartbeater.cc:502] Master 127.0.107.126:40235 was elected leader, sending a full tablet report...
I20250114 02:55:10.908380 3736 consensus_queue.cc:260] T c174bcbc5d51402b8ae332b87ffa2139 P 39c64942315c4716baaee0c251c05acd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:10.922179 3613 tablet_service.cc:1514] Processing DeleteTablet for tablet 55ce9a7d0c0f45bf982b5c064ee1f676 with delete_type TABLET_DATA_DELETED (replaced by tablet c174bcbc5d51402b8ae332b87ffa2139) from {username='slave'} at 127.0.0.1:60674
W20250114 02:55:10.928071 2969 catalog_manager.cc:4852] TS 5017c179ddc640fc9ebf5dea7f3957f8 (127.0.107.67:45409): delete failed for tablet 55ce9a7d0c0f45bf982b5c064ee1f676 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 55ce9a7d0c0f45bf982b5c064ee1f676 already in progress: creating tablet
I20250114 02:55:10.930099 3738 raft_consensus.cc:357] T 55ce9a7d0c0f45bf982b5c064ee1f676 P 7b7de0f9038c4ba287750ad132cb3597 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } } peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } }
I20250114 02:55:10.931088 3739 raft_consensus.cc:357] T 55ce9a7d0c0f45bf982b5c064ee1f676 P 5017c179ddc640fc9ebf5dea7f3957f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } } peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } }
I20250114 02:55:10.932060 3739 raft_consensus.cc:383] T 55ce9a7d0c0f45bf982b5c064ee1f676 P 5017c179ddc640fc9ebf5dea7f3957f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:55:10.932494 3738 raft_consensus.cc:383] T 55ce9a7d0c0f45bf982b5c064ee1f676 P 7b7de0f9038c4ba287750ad132cb3597 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:55:10.932821 3739 raft_consensus.cc:738] T 55ce9a7d0c0f45bf982b5c064ee1f676 P 5017c179ddc640fc9ebf5dea7f3957f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5017c179ddc640fc9ebf5dea7f3957f8, State: Initialized, Role: FOLLOWER
I20250114 02:55:10.933241 3738 raft_consensus.cc:738] T 55ce9a7d0c0f45bf982b5c064ee1f676 P 7b7de0f9038c4ba287750ad132cb3597 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b7de0f9038c4ba287750ad132cb3597, State: Initialized, Role: FOLLOWER
I20250114 02:55:10.934129 3739 consensus_queue.cc:260] T 55ce9a7d0c0f45bf982b5c064ee1f676 P 5017c179ddc640fc9ebf5dea7f3957f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } } peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } }
I20250114 02:55:10.935595 3738 consensus_queue.cc:260] T 55ce9a7d0c0f45bf982b5c064ee1f676 P 7b7de0f9038c4ba287750ad132cb3597 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } } peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } }
I20250114 02:55:10.945751 3736 ts_tablet_manager.cc:1428] T c174bcbc5d51402b8ae332b87ffa2139 P 39c64942315c4716baaee0c251c05acd: Time spent starting tablet: real 0.043s user 0.005s sys 0.000s
I20250114 02:55:10.946830 2970 catalog_manager.cc:4872] TS 39c64942315c4716baaee0c251c05acd (127.0.107.66:33487): tablet 55ce9a7d0c0f45bf982b5c064ee1f676 (table default.table_bab0c0670ef34820b254580eeb9eef70 [id=89d59ce4ef604cce9b1e9a01a4aea791]) successfully deleted
I20250114 02:55:10.954459 3739 ts_tablet_manager.cc:1428] T 55ce9a7d0c0f45bf982b5c064ee1f676 P 5017c179ddc640fc9ebf5dea7f3957f8: Time spent starting tablet: real 0.106s user 0.038s sys 0.009s
I20250114 02:55:10.955574 3739 tablet_bootstrap.cc:492] T c174bcbc5d51402b8ae332b87ffa2139 P 5017c179ddc640fc9ebf5dea7f3957f8: Bootstrap starting.
W20250114 02:55:10.958456 3541 tablet.cc:2367] T c174bcbc5d51402b8ae332b87ffa2139 P 39c64942315c4716baaee0c251c05acd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250114 02:55:10.963811 3613 tablet_service.cc:1514] Processing DeleteTablet for tablet 55ce9a7d0c0f45bf982b5c064ee1f676 with delete_type TABLET_DATA_DELETED (replaced by tablet c174bcbc5d51402b8ae332b87ffa2139) from {username='slave'} at 127.0.0.1:60674
I20250114 02:55:10.964728 3741 tablet_replica.cc:331] T 55ce9a7d0c0f45bf982b5c064ee1f676 P 5017c179ddc640fc9ebf5dea7f3957f8: stopping tablet replica
I20250114 02:55:10.965620 3741 raft_consensus.cc:2238] T 55ce9a7d0c0f45bf982b5c064ee1f676 P 5017c179ddc640fc9ebf5dea7f3957f8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250114 02:55:10.966181 3741 raft_consensus.cc:2267] T 55ce9a7d0c0f45bf982b5c064ee1f676 P 5017c179ddc640fc9ebf5dea7f3957f8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250114 02:55:10.971483 3398 heartbeater.cc:502] Master 127.0.107.126:40235 was elected leader, sending a full tablet report...
I20250114 02:55:10.982200 3741 ts_tablet_manager.cc:1905] T 55ce9a7d0c0f45bf982b5c064ee1f676 P 5017c179ddc640fc9ebf5dea7f3957f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:55:10.986186 2979 master_service.cc:946] ReplaceTablet: received request to replace tablet c174bcbc5d51402b8ae332b87ffa2139 from {username='slave'} at 127.0.0.1:57960
I20250114 02:55:10.993108 3738 ts_tablet_manager.cc:1428] T 55ce9a7d0c0f45bf982b5c064ee1f676 P 7b7de0f9038c4ba287750ad132cb3597: Time spent starting tablet: real 0.194s user 0.037s sys 0.012s
I20250114 02:55:11.005203 429 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrCUXNq/build/tsan/bin/kudu
/tmp/dist-test-taskrCUXNq/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.107.124:44431
--webserver_interface=127.0.107.124
--webserver_port=36967
--builtin_ntp_servers=127.0.107.84:38241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.107.126:40235,127.0.107.125:38653,127.0.107.124:44431
--hive_metastore_uris=thrift://127.0.0.1:33943
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250114 02:55:10.995672 3326 tablet_service.cc:1514] Processing DeleteTablet for tablet 55ce9a7d0c0f45bf982b5c064ee1f676 with delete_type TABLET_DATA_DELETED (replaced by tablet c174bcbc5d51402b8ae332b87ffa2139) from {username='slave'} at 127.0.0.1:58826
W20250114 02:55:11.009311 2969 catalog_manager.cc:4852] TS 7b7de0f9038c4ba287750ad132cb3597 (127.0.107.65:34329): delete failed for tablet 55ce9a7d0c0f45bf982b5c064ee1f676 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 55ce9a7d0c0f45bf982b5c064ee1f676 already in progress: creating tablet
I20250114 02:55:11.015300 3326 tablet_service.cc:1514] Processing DeleteTablet for tablet 55ce9a7d0c0f45bf982b5c064ee1f676 with delete_type TABLET_DATA_DELETED (replaced by tablet c174bcbc5d51402b8ae332b87ffa2139) from {username='slave'} at 127.0.0.1:58826
I20250114 02:55:11.016216 3742 tablet_replica.cc:331] T 55ce9a7d0c0f45bf982b5c064ee1f676 P 7b7de0f9038c4ba287750ad132cb3597: stopping tablet replica
I20250114 02:55:11.017045 3742 raft_consensus.cc:2238] T 55ce9a7d0c0f45bf982b5c064ee1f676 P 7b7de0f9038c4ba287750ad132cb3597 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250114 02:55:11.017562 3742 raft_consensus.cc:2267] T 55ce9a7d0c0f45bf982b5c064ee1f676 P 7b7de0f9038c4ba287750ad132cb3597 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250114 02:55:11.024855 2979 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27: Sending DeleteTablet for 3 replicas of tablet c174bcbc5d51402b8ae332b87ffa2139
I20250114 02:55:11.026211 2979 catalog_manager.cc:6596] ReplaceTablet: tablet c174bcbc5d51402b8ae332b87ffa2139 deleted and replaced by tablet 91303b0646af43409c3ad051facf590e
I20250114 02:55:11.026537 3326 tablet_service.cc:1514] Processing DeleteTablet for tablet c174bcbc5d51402b8ae332b87ffa2139 with delete_type TABLET_DATA_DELETED (replaced by tablet 91303b0646af43409c3ad051facf590e) from {username='slave'} at 127.0.0.1:58826
I20250114 02:55:11.027158 3742 ts_tablet_manager.cc:1905] T 55ce9a7d0c0f45bf982b5c064ee1f676 P 7b7de0f9038c4ba287750ad132cb3597: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:55:11.027396 3470 tablet_service.cc:1514] Processing DeleteTablet for tablet c174bcbc5d51402b8ae332b87ffa2139 with delete_type TABLET_DATA_DELETED (replaced by tablet 91303b0646af43409c3ad051facf590e) from {username='slave'} at 127.0.0.1:58610
I20250114 02:55:11.028330 3743 tablet_replica.cc:331] T c174bcbc5d51402b8ae332b87ffa2139 P 39c64942315c4716baaee0c251c05acd: stopping tablet replica
I20250114 02:55:11.028972 3743 raft_consensus.cc:2238] T c174bcbc5d51402b8ae332b87ffa2139 P 39c64942315c4716baaee0c251c05acd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250114 02:55:11.029415 3743 raft_consensus.cc:2267] T c174bcbc5d51402b8ae332b87ffa2139 P 39c64942315c4716baaee0c251c05acd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250114 02:55:11.030923 3613 tablet_service.cc:1514] Processing DeleteTablet for tablet c174bcbc5d51402b8ae332b87ffa2139 with delete_type TABLET_DATA_DELETED (replaced by tablet 91303b0646af43409c3ad051facf590e) from {username='slave'} at 127.0.0.1:60674
I20250114 02:55:11.032995 3743 ts_tablet_manager.cc:1905] T c174bcbc5d51402b8ae332b87ffa2139 P 39c64942315c4716baaee0c251c05acd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:55:11.047132 3738 tablet_bootstrap.cc:492] T c174bcbc5d51402b8ae332b87ffa2139 P 7b7de0f9038c4ba287750ad132cb3597: Bootstrap starting.
I20250114 02:55:11.060756 3743 ts_tablet_manager.cc:1918] T c174bcbc5d51402b8ae332b87ffa2139 P 39c64942315c4716baaee0c251c05acd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:55:11.061146 3739 tablet_bootstrap.cc:654] T c174bcbc5d51402b8ae332b87ffa2139 P 5017c179ddc640fc9ebf5dea7f3957f8: Neither blocks nor log segments found. Creating new log.
I20250114 02:55:11.061324 3743 log.cc:1198] T c174bcbc5d51402b8ae332b87ffa2139 P 39c64942315c4716baaee0c251c05acd: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/c174bcbc5d51402b8ae332b87ffa2139
I20250114 02:55:11.062111 3743 ts_tablet_manager.cc:1939] T c174bcbc5d51402b8ae332b87ffa2139 P 39c64942315c4716baaee0c251c05acd: Deleting consensus metadata
I20250114 02:55:11.064747 2970 catalog_manager.cc:4872] TS 39c64942315c4716baaee0c251c05acd (127.0.107.66:33487): tablet c174bcbc5d51402b8ae332b87ffa2139 (table default.table_bab0c0670ef34820b254580eeb9eef70 [id=89d59ce4ef604cce9b1e9a01a4aea791]) successfully deleted
I20250114 02:55:11.065378 2984 catalog_manager.cc:2149] created HMS catalog entry for table [id=0b4964a250114a8f855dddd9ba3b6f76]
I20250114 02:55:11.073056 2981 master_service.cc:946] ReplaceTablet: received request to replace tablet 55ce9a7d0c0f45bf982b5c064ee1f676 from {username='slave'} at 127.0.0.1:57960
I20250114 02:55:11.088826 3738 tablet_bootstrap.cc:654] T c174bcbc5d51402b8ae332b87ffa2139 P 7b7de0f9038c4ba287750ad132cb3597: Neither blocks nor log segments found. Creating new log.
I20250114 02:55:11.090179 2984 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:57960:
table { table_name: "default.Table_b4cf0d29abe14186b59a52f5bcf7d11d" } new_table_name: "default.Table_abbbf6099c5b4880ad06b2e250de4248" modify_external_catalogs: true
I20250114 02:55:11.092360 3741 ts_tablet_manager.cc:1918] T 55ce9a7d0c0f45bf982b5c064ee1f676 P 5017c179ddc640fc9ebf5dea7f3957f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:55:11.100127 3326 tablet_service.cc:1467] Processing CreateTablet for tablet 91303b0646af43409c3ad051facf590e (DEFAULT_TABLE table=default.table_bab0c0670ef34820b254580eeb9eef70 [id=89d59ce4ef604cce9b1e9a01a4aea791]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:55:11.105126 3326 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91303b0646af43409c3ad051facf590e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:55:11.105439 3470 tablet_service.cc:1467] Processing CreateTablet for tablet 91303b0646af43409c3ad051facf590e (DEFAULT_TABLE table=default.table_bab0c0670ef34820b254580eeb9eef70 [id=89d59ce4ef604cce9b1e9a01a4aea791]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:55:11.106525 3470 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91303b0646af43409c3ad051facf590e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:55:11.105547 3613 tablet_service.cc:1467] Processing CreateTablet for tablet 91303b0646af43409c3ad051facf590e (DEFAULT_TABLE table=default.table_bab0c0670ef34820b254580eeb9eef70 [id=89d59ce4ef604cce9b1e9a01a4aea791]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:55:11.117515 3613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91303b0646af43409c3ad051facf590e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:55:11.127416 3736 tablet_bootstrap.cc:492] T 91303b0646af43409c3ad051facf590e P 39c64942315c4716baaee0c251c05acd: Bootstrap starting.
I20250114 02:55:11.128410 3741 log.cc:1198] T 55ce9a7d0c0f45bf982b5c064ee1f676 P 5017c179ddc640fc9ebf5dea7f3957f8: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/55ce9a7d0c0f45bf982b5c064ee1f676
I20250114 02:55:11.129362 3741 ts_tablet_manager.cc:1939] T 55ce9a7d0c0f45bf982b5c064ee1f676 P 5017c179ddc640fc9ebf5dea7f3957f8: Deleting consensus metadata
I20250114 02:55:11.134146 3471 tablet_service.cc:1467] Processing CreateTablet for tablet 2e2bcc70bcab4dfb8e0d8835dbf92749 (DEFAULT_TABLE table=default.table_b4cf0d29abe14186b59a52f5bcf7d11d [id=0b4964a250114a8f855dddd9ba3b6f76]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:55:11.135103 3739 tablet_bootstrap.cc:492] T c174bcbc5d51402b8ae332b87ffa2139 P 5017c179ddc640fc9ebf5dea7f3957f8: No bootstrap required, opened a new log
I20250114 02:55:11.135283 3471 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e2bcc70bcab4dfb8e0d8835dbf92749. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:55:11.135540 2969 catalog_manager.cc:4872] TS 5017c179ddc640fc9ebf5dea7f3957f8 (127.0.107.67:45409): tablet 55ce9a7d0c0f45bf982b5c064ee1f676 (table default.table_bab0c0670ef34820b254580eeb9eef70 [id=89d59ce4ef604cce9b1e9a01a4aea791]) successfully deleted
I20250114 02:55:11.136039 3739 ts_tablet_manager.cc:1397] T c174bcbc5d51402b8ae332b87ffa2139 P 5017c179ddc640fc9ebf5dea7f3957f8: Time spent bootstrapping tablet: real 0.181s user 0.008s sys 0.004s
I20250114 02:55:11.138068 3612 tablet_service.cc:1467] Processing CreateTablet for tablet 2e2bcc70bcab4dfb8e0d8835dbf92749 (DEFAULT_TABLE table=default.table_b4cf0d29abe14186b59a52f5bcf7d11d [id=0b4964a250114a8f855dddd9ba3b6f76]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:55:11.139259 3612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e2bcc70bcab4dfb8e0d8835dbf92749. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:55:11.142369 2982 catalog_manager.cc:2149] created HMS catalog entry for table [id=5dbcc4cbd424444490d042e890c01b93]
W20250114 02:55:11.145675 2969 catalog_manager.cc:4852] TS 5017c179ddc640fc9ebf5dea7f3957f8 (127.0.107.67:45409): delete failed for tablet c174bcbc5d51402b8ae332b87ffa2139 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c174bcbc5d51402b8ae332b87ffa2139 already in progress: creating tablet
I20250114 02:55:11.147960 3327 tablet_service.cc:1467] Processing CreateTablet for tablet 2e2bcc70bcab4dfb8e0d8835dbf92749 (DEFAULT_TABLE table=default.table_b4cf0d29abe14186b59a52f5bcf7d11d [id=0b4964a250114a8f855dddd9ba3b6f76]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:55:11.149055 3736 tablet_bootstrap.cc:654] T 91303b0646af43409c3ad051facf590e P 39c64942315c4716baaee0c251c05acd: Neither blocks nor log segments found. Creating new log.
I20250114 02:55:11.149138 3327 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e2bcc70bcab4dfb8e0d8835dbf92749. 1 dirs total, 0 dirs full, 0 dirs failed
F20250114 02:55:11.153823 3213 hms_notification_log_listener.cc:263] Received out-of-order notification log event (last processed event ID: 1): 1 CREATE_TABLE default.table_627b0faab8eb44c58814768e6dd28c15
*** Check failure stack trace: ***
I20250114 02:55:11.160917 3742 ts_tablet_manager.cc:1918] T 55ce9a7d0c0f45bf982b5c064ee1f676 P 7b7de0f9038c4ba287750ad132cb3597: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:55:11.161526 3742 log.cc:1198] T 55ce9a7d0c0f45bf982b5c064ee1f676 P 7b7de0f9038c4ba287750ad132cb3597: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/55ce9a7d0c0f45bf982b5c064ee1f676
I20250114 02:55:11.162770 3742 ts_tablet_manager.cc:1939] T 55ce9a7d0c0f45bf982b5c064ee1f676 P 7b7de0f9038c4ba287750ad132cb3597: Deleting consensus metadata
I20250114 02:55:11.165977 3736 tablet_bootstrap.cc:492] T 91303b0646af43409c3ad051facf590e P 39c64942315c4716baaee0c251c05acd: No bootstrap required, opened a new log
I20250114 02:55:11.166049 2969 catalog_manager.cc:4872] TS 7b7de0f9038c4ba287750ad132cb3597 (127.0.107.65:34329): tablet 55ce9a7d0c0f45bf982b5c064ee1f676 (table default.table_bab0c0670ef34820b254580eeb9eef70 [id=89d59ce4ef604cce9b1e9a01a4aea791]) successfully deleted
I20250114 02:55:11.167017 3736 ts_tablet_manager.cc:1397] T 91303b0646af43409c3ad051facf590e P 39c64942315c4716baaee0c251c05acd: Time spent bootstrapping tablet: real 0.040s user 0.009s sys 0.003s
W20250114 02:55:11.167109 2969 catalog_manager.cc:4852] TS 7b7de0f9038c4ba287750ad132cb3597 (127.0.107.65:34329): delete failed for tablet c174bcbc5d51402b8ae332b87ffa2139 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c174bcbc5d51402b8ae332b87ffa2139 already in progress: creating tablet
I20250114 02:55:11.170969 2982 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57960:
table { table_name: "default.Table_8aa1373bd9824b1a8cbed27ede045e55" } modify_external_catalogs: true
I20250114 02:55:11.171849 2982 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57960:
table { table_name: "default.Table_8aa1373bd9824b1a8cbed27ede045e55" } modify_external_catalogs: true
I20250114 02:55:11.172904 3736 raft_consensus.cc:357] T 91303b0646af43409c3ad051facf590e P 39c64942315c4716baaee0c251c05acd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:11.173846 3736 raft_consensus.cc:383] T 91303b0646af43409c3ad051facf590e P 39c64942315c4716baaee0c251c05acd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:55:11.174163 3736 raft_consensus.cc:738] T 91303b0646af43409c3ad051facf590e P 39c64942315c4716baaee0c251c05acd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39c64942315c4716baaee0c251c05acd, State: Initialized, Role: FOLLOWER
I20250114 02:55:11.174960 3736 consensus_queue.cc:260] T 91303b0646af43409c3ad051facf590e P 39c64942315c4716baaee0c251c05acd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:11.179549 3736 ts_tablet_manager.cc:1428] T 91303b0646af43409c3ad051facf590e P 39c64942315c4716baaee0c251c05acd: Time spent starting tablet: real 0.012s user 0.003s sys 0.003s
I20250114 02:55:11.182395 3736 tablet_bootstrap.cc:492] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 39c64942315c4716baaee0c251c05acd: Bootstrap starting.
I20250114 02:55:11.188864 3736 tablet_bootstrap.cc:654] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 39c64942315c4716baaee0c251c05acd: Neither blocks nor log segments found. Creating new log.
I20250114 02:55:11.194564 3736 tablet_bootstrap.cc:492] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 39c64942315c4716baaee0c251c05acd: No bootstrap required, opened a new log
I20250114 02:55:11.195129 3736 ts_tablet_manager.cc:1397] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 39c64942315c4716baaee0c251c05acd: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.000s
I20250114 02:55:11.141834 3739 raft_consensus.cc:357] T c174bcbc5d51402b8ae332b87ffa2139 P 5017c179ddc640fc9ebf5dea7f3957f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:11.197480 3739 raft_consensus.cc:383] T c174bcbc5d51402b8ae332b87ffa2139 P 5017c179ddc640fc9ebf5dea7f3957f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:55:11.197775 3739 raft_consensus.cc:738] T c174bcbc5d51402b8ae332b87ffa2139 P 5017c179ddc640fc9ebf5dea7f3957f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5017c179ddc640fc9ebf5dea7f3957f8, State: Initialized, Role: FOLLOWER
I20250114 02:55:11.197944 3736 raft_consensus.cc:357] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 39c64942315c4716baaee0c251c05acd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:11.198815 3736 raft_consensus.cc:383] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 39c64942315c4716baaee0c251c05acd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:55:11.198460 3739 consensus_queue.cc:260] T c174bcbc5d51402b8ae332b87ffa2139 P 5017c179ddc640fc9ebf5dea7f3957f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:11.199156 3736 raft_consensus.cc:738] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 39c64942315c4716baaee0c251c05acd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39c64942315c4716baaee0c251c05acd, State: Initialized, Role: FOLLOWER
I20250114 02:55:11.199962 3736 consensus_queue.cc:260] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 39c64942315c4716baaee0c251c05acd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:11.205862 3736 ts_tablet_manager.cc:1428] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 39c64942315c4716baaee0c251c05acd: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20250114 02:55:11.210427 3738 tablet_bootstrap.cc:492] T c174bcbc5d51402b8ae332b87ffa2139 P 7b7de0f9038c4ba287750ad132cb3597: No bootstrap required, opened a new log
I20250114 02:55:11.210836 3738 ts_tablet_manager.cc:1397] T c174bcbc5d51402b8ae332b87ffa2139 P 7b7de0f9038c4ba287750ad132cb3597: Time spent bootstrapping tablet: real 0.164s user 0.007s sys 0.003s
I20250114 02:55:11.212980 3738 raft_consensus.cc:357] T c174bcbc5d51402b8ae332b87ffa2139 P 7b7de0f9038c4ba287750ad132cb3597 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:11.213508 3738 raft_consensus.cc:383] T c174bcbc5d51402b8ae332b87ffa2139 P 7b7de0f9038c4ba287750ad132cb3597 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:55:11.213732 3738 raft_consensus.cc:738] T c174bcbc5d51402b8ae332b87ffa2139 P 7b7de0f9038c4ba287750ad132cb3597 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b7de0f9038c4ba287750ad132cb3597, State: Initialized, Role: FOLLOWER
I20250114 02:55:11.214378 3738 consensus_queue.cc:260] T c174bcbc5d51402b8ae332b87ffa2139 P 7b7de0f9038c4ba287750ad132cb3597 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
W20250114 02:55:11.214754 2981 catalog_manager.cc:5359] Ignoring report from unknown tablet c174bcbc5d51402b8ae332b87ffa2139
I20250114 02:55:11.201174 3739 ts_tablet_manager.cc:1428] T c174bcbc5d51402b8ae332b87ffa2139 P 5017c179ddc640fc9ebf5dea7f3957f8: Time spent starting tablet: real 0.065s user 0.005s sys 0.000s
I20250114 02:55:11.224366 3738 ts_tablet_manager.cc:1428] T c174bcbc5d51402b8ae332b87ffa2139 P 7b7de0f9038c4ba287750ad132cb3597: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20250114 02:55:11.225339 3738 tablet_bootstrap.cc:492] T 91303b0646af43409c3ad051facf590e P 7b7de0f9038c4ba287750ad132cb3597: Bootstrap starting.
W20250114 02:55:11.226161 2978 catalog_manager.cc:5359] Ignoring report from unknown tablet c174bcbc5d51402b8ae332b87ffa2139
I20250114 02:55:11.228372 3739 tablet_bootstrap.cc:492] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 5017c179ddc640fc9ebf5dea7f3957f8: Bootstrap starting.
W20250114 02:55:11.229849 2979 catalog_manager.cc:5359] Ignoring report from unknown tablet c174bcbc5d51402b8ae332b87ffa2139
W20250114 02:55:11.234287 2978 catalog_manager.cc:5359] Ignoring report from unknown tablet c174bcbc5d51402b8ae332b87ffa2139
I20250114 02:55:11.234608 3327 tablet_service.cc:1467] Processing CreateTablet for tablet 688cd8217fb04e4e930cb2421bfcfd9b (DEFAULT_TABLE table=default.table_8aa1373bd9824b1a8cbed27ede045e55 [id=5dbcc4cbd424444490d042e890c01b93]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:55:11.235671 3327 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 688cd8217fb04e4e930cb2421bfcfd9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:55:11.235993 3471 tablet_service.cc:1467] Processing CreateTablet for tablet 688cd8217fb04e4e930cb2421bfcfd9b (DEFAULT_TABLE table=default.table_8aa1373bd9824b1a8cbed27ede045e55 [id=5dbcc4cbd424444490d042e890c01b93]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:55:11.236515 3613 tablet_service.cc:1467] Processing CreateTablet for tablet 688cd8217fb04e4e930cb2421bfcfd9b (DEFAULT_TABLE table=default.table_8aa1373bd9824b1a8cbed27ede045e55 [id=5dbcc4cbd424444490d042e890c01b93]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:55:11.237046 3471 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 688cd8217fb04e4e930cb2421bfcfd9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:55:11.237589 3613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 688cd8217fb04e4e930cb2421bfcfd9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:55:11.237643 3739 tablet_bootstrap.cc:654] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 5017c179ddc640fc9ebf5dea7f3957f8: Neither blocks nor log segments found. Creating new log.
I20250114 02:55:11.248720 3739 tablet_bootstrap.cc:492] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 5017c179ddc640fc9ebf5dea7f3957f8: No bootstrap required, opened a new log
I20250114 02:55:11.249295 3739 ts_tablet_manager.cc:1397] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 5017c179ddc640fc9ebf5dea7f3957f8: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.013s
I20250114 02:55:11.251986 3738 tablet_bootstrap.cc:654] T 91303b0646af43409c3ad051facf590e P 7b7de0f9038c4ba287750ad132cb3597: Neither blocks nor log segments found. Creating new log.
I20250114 02:55:11.252844 3739 raft_consensus.cc:357] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 5017c179ddc640fc9ebf5dea7f3957f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:11.253747 3739 raft_consensus.cc:383] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 5017c179ddc640fc9ebf5dea7f3957f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:55:11.253973 3736 tablet_bootstrap.cc:492] T 688cd8217fb04e4e930cb2421bfcfd9b P 39c64942315c4716baaee0c251c05acd: Bootstrap starting.
I20250114 02:55:11.254122 3739 raft_consensus.cc:738] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 5017c179ddc640fc9ebf5dea7f3957f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5017c179ddc640fc9ebf5dea7f3957f8, State: Initialized, Role: FOLLOWER
I20250114 02:55:11.254835 3739 consensus_queue.cc:260] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 5017c179ddc640fc9ebf5dea7f3957f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:11.259936 3736 tablet_bootstrap.cc:654] T 688cd8217fb04e4e930cb2421bfcfd9b P 39c64942315c4716baaee0c251c05acd: Neither blocks nor log segments found. Creating new log.
I20250114 02:55:11.261955 2979 master_service.cc:946] ReplaceTablet: received request to replace tablet c174bcbc5d51402b8ae332b87ffa2139 from {username='slave'} at 127.0.0.1:57960
I20250114 02:55:11.271363 3739 ts_tablet_manager.cc:1428] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 5017c179ddc640fc9ebf5dea7f3957f8: Time spent starting tablet: real 0.022s user 0.004s sys 0.002s
@ 0x7fb90bf6a950 google::LogMessage::Flush() at ??:0
I20250114 02:55:11.283711 2979 master_service.cc:946] ReplaceTablet: received request to replace tablet c174bcbc5d51402b8ae332b87ffa2139 from {username='slave'} at 127.0.0.1:57960
I20250114 02:55:11.290642 2978 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57960:
name: "default.Table_f39207a74e3a4f68acdb67e4a70f7510"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:55:11.289935 3739 tablet_bootstrap.cc:492] T 91303b0646af43409c3ad051facf590e P 5017c179ddc640fc9ebf5dea7f3957f8: Bootstrap starting.
@ 0x7fb90bf6a6bf google::LogMessage::~LogMessage() at ??:0
I20250114 02:55:11.298048 3739 tablet_bootstrap.cc:654] T 91303b0646af43409c3ad051facf590e P 5017c179ddc640fc9ebf5dea7f3957f8: Neither blocks nor log segments found. Creating new log.
I20250114 02:55:11.302640 3738 tablet_bootstrap.cc:492] T 91303b0646af43409c3ad051facf590e P 7b7de0f9038c4ba287750ad132cb3597: No bootstrap required, opened a new log
I20250114 02:55:11.303182 3738 ts_tablet_manager.cc:1397] T 91303b0646af43409c3ad051facf590e P 7b7de0f9038c4ba287750ad132cb3597: Time spent bootstrapping tablet: real 0.078s user 0.008s sys 0.002s
I20250114 02:55:11.306834 3739 tablet_bootstrap.cc:492] T 91303b0646af43409c3ad051facf590e P 5017c179ddc640fc9ebf5dea7f3957f8: No bootstrap required, opened a new log
I20250114 02:55:11.306306 3738 raft_consensus.cc:357] T 91303b0646af43409c3ad051facf590e P 7b7de0f9038c4ba287750ad132cb3597 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:11.307257 3739 ts_tablet_manager.cc:1397] T 91303b0646af43409c3ad051facf590e P 5017c179ddc640fc9ebf5dea7f3957f8: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.007s
I20250114 02:55:11.307269 3738 raft_consensus.cc:383] T 91303b0646af43409c3ad051facf590e P 7b7de0f9038c4ba287750ad132cb3597 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:55:11.307618 3738 raft_consensus.cc:738] T 91303b0646af43409c3ad051facf590e P 7b7de0f9038c4ba287750ad132cb3597 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b7de0f9038c4ba287750ad132cb3597, State: Initialized, Role: FOLLOWER
I20250114 02:55:11.310015 3739 raft_consensus.cc:357] T 91303b0646af43409c3ad051facf590e P 5017c179ddc640fc9ebf5dea7f3957f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:11.310791 3739 raft_consensus.cc:383] T 91303b0646af43409c3ad051facf590e P 5017c179ddc640fc9ebf5dea7f3957f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:55:11.311089 3739 raft_consensus.cc:738] T 91303b0646af43409c3ad051facf590e P 5017c179ddc640fc9ebf5dea7f3957f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5017c179ddc640fc9ebf5dea7f3957f8, State: Initialized, Role: FOLLOWER
I20250114 02:55:11.311913 3739 consensus_queue.cc:260] T 91303b0646af43409c3ad051facf590e P 5017c179ddc640fc9ebf5dea7f3957f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:11.318764 3736 tablet_bootstrap.cc:492] T 688cd8217fb04e4e930cb2421bfcfd9b P 39c64942315c4716baaee0c251c05acd: No bootstrap required, opened a new log
I20250114 02:55:11.318450 3738 consensus_queue.cc:260] T 91303b0646af43409c3ad051facf590e P 7b7de0f9038c4ba287750ad132cb3597 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:11.319170 3736 ts_tablet_manager.cc:1397] T 688cd8217fb04e4e930cb2421bfcfd9b P 39c64942315c4716baaee0c251c05acd: Time spent bootstrapping tablet: real 0.065s user 0.025s sys 0.027s
I20250114 02:55:11.322439 3736 raft_consensus.cc:357] T 688cd8217fb04e4e930cb2421bfcfd9b P 39c64942315c4716baaee0c251c05acd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:11.323098 3736 raft_consensus.cc:383] T 688cd8217fb04e4e930cb2421bfcfd9b P 39c64942315c4716baaee0c251c05acd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:55:11.323359 3736 raft_consensus.cc:738] T 688cd8217fb04e4e930cb2421bfcfd9b P 39c64942315c4716baaee0c251c05acd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39c64942315c4716baaee0c251c05acd, State: Initialized, Role: FOLLOWER
I20250114 02:55:11.324011 3736 consensus_queue.cc:260] T 688cd8217fb04e4e930cb2421bfcfd9b P 39c64942315c4716baaee0c251c05acd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:11.328725 3738 ts_tablet_manager.cc:1428] T 91303b0646af43409c3ad051facf590e P 7b7de0f9038c4ba287750ad132cb3597: Time spent starting tablet: real 0.025s user 0.009s sys 0.001s
I20250114 02:55:11.329684 3738 tablet_bootstrap.cc:492] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 7b7de0f9038c4ba287750ad132cb3597: Bootstrap starting.
I20250114 02:55:11.332520 3739 ts_tablet_manager.cc:1428] T 91303b0646af43409c3ad051facf590e P 5017c179ddc640fc9ebf5dea7f3957f8: Time spent starting tablet: real 0.025s user 0.007s sys 0.001s
I20250114 02:55:11.335887 3738 tablet_bootstrap.cc:654] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 7b7de0f9038c4ba287750ad132cb3597: Neither blocks nor log segments found. Creating new log.
I20250114 02:55:11.338364 3736 ts_tablet_manager.cc:1428] T 688cd8217fb04e4e930cb2421bfcfd9b P 39c64942315c4716baaee0c251c05acd: Time spent starting tablet: real 0.019s user 0.003s sys 0.004s
I20250114 02:55:11.357874 3739 tablet_bootstrap.cc:492] T 688cd8217fb04e4e930cb2421bfcfd9b P 5017c179ddc640fc9ebf5dea7f3957f8: Bootstrap starting.
W20250114 02:55:11.360800 3399 tablet.cc:2367] T 91303b0646af43409c3ad051facf590e P 7b7de0f9038c4ba287750ad132cb3597: Can't schedule compaction. Clean time has not been advanced past its initial value.
@ 0x7fb916e6482a kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250114 02:55:11.366814 3739 tablet_bootstrap.cc:654] T 688cd8217fb04e4e930cb2421bfcfd9b P 5017c179ddc640fc9ebf5dea7f3957f8: Neither blocks nor log segments found. Creating new log.
I20250114 02:55:11.371939 2981 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57960:
name: "default.Table_c27b5989e4ba41f285a14120cb0dba75"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:55:11.374801 3738 tablet_bootstrap.cc:492] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 7b7de0f9038c4ba287750ad132cb3597: No bootstrap required, opened a new log
I20250114 02:55:11.375231 3738 ts_tablet_manager.cc:1397] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 7b7de0f9038c4ba287750ad132cb3597: Time spent bootstrapping tablet: real 0.046s user 0.024s sys 0.016s
I20250114 02:55:11.377753 3738 raft_consensus.cc:357] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 7b7de0f9038c4ba287750ad132cb3597 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:11.378424 3738 raft_consensus.cc:383] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 7b7de0f9038c4ba287750ad132cb3597 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:55:11.378719 3738 raft_consensus.cc:738] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 7b7de0f9038c4ba287750ad132cb3597 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b7de0f9038c4ba287750ad132cb3597, State: Initialized, Role: FOLLOWER
I20250114 02:55:11.379421 3738 consensus_queue.cc:260] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 7b7de0f9038c4ba287750ad132cb3597 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:11.382045 3738 ts_tablet_manager.cc:1428] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 7b7de0f9038c4ba287750ad132cb3597: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250114 02:55:11.382918 3738 tablet_bootstrap.cc:492] T 688cd8217fb04e4e930cb2421bfcfd9b P 7b7de0f9038c4ba287750ad132cb3597: Bootstrap starting.
I20250114 02:55:11.388939 3738 tablet_bootstrap.cc:654] T 688cd8217fb04e4e930cb2421bfcfd9b P 7b7de0f9038c4ba287750ad132cb3597: Neither blocks nor log segments found. Creating new log.
W20250114 02:55:11.390175 3685 tablet.cc:2367] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 5017c179ddc640fc9ebf5dea7f3957f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250114 02:55:11.397046 3739 tablet_bootstrap.cc:492] T 688cd8217fb04e4e930cb2421bfcfd9b P 5017c179ddc640fc9ebf5dea7f3957f8: No bootstrap required, opened a new log
I20250114 02:55:11.397490 3739 ts_tablet_manager.cc:1397] T 688cd8217fb04e4e930cb2421bfcfd9b P 5017c179ddc640fc9ebf5dea7f3957f8: Time spent bootstrapping tablet: real 0.040s user 0.013s sys 0.014s
I20250114 02:55:11.400086 3739 raft_consensus.cc:357] T 688cd8217fb04e4e930cb2421bfcfd9b P 5017c179ddc640fc9ebf5dea7f3957f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:11.400830 3739 raft_consensus.cc:383] T 688cd8217fb04e4e930cb2421bfcfd9b P 5017c179ddc640fc9ebf5dea7f3957f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:55:11.401080 3739 raft_consensus.cc:738] T 688cd8217fb04e4e930cb2421bfcfd9b P 5017c179ddc640fc9ebf5dea7f3957f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5017c179ddc640fc9ebf5dea7f3957f8, State: Initialized, Role: FOLLOWER
I20250114 02:55:11.401738 3739 consensus_queue.cc:260] T 688cd8217fb04e4e930cb2421bfcfd9b P 5017c179ddc640fc9ebf5dea7f3957f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:11.402271 3738 tablet_bootstrap.cc:492] T 688cd8217fb04e4e930cb2421bfcfd9b P 7b7de0f9038c4ba287750ad132cb3597: No bootstrap required, opened a new log
I20250114 02:55:11.402673 3738 ts_tablet_manager.cc:1397] T 688cd8217fb04e4e930cb2421bfcfd9b P 7b7de0f9038c4ba287750ad132cb3597: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.004s
I20250114 02:55:11.405130 3738 raft_consensus.cc:357] T 688cd8217fb04e4e930cb2421bfcfd9b P 7b7de0f9038c4ba287750ad132cb3597 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:11.405774 3738 raft_consensus.cc:383] T 688cd8217fb04e4e930cb2421bfcfd9b P 7b7de0f9038c4ba287750ad132cb3597 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:55:11.406162 3738 raft_consensus.cc:738] T 688cd8217fb04e4e930cb2421bfcfd9b P 7b7de0f9038c4ba287750ad132cb3597 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b7de0f9038c4ba287750ad132cb3597, State: Initialized, Role: FOLLOWER
I20250114 02:55:11.406899 3738 consensus_queue.cc:260] T 688cd8217fb04e4e930cb2421bfcfd9b P 7b7de0f9038c4ba287750ad132cb3597 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:11.411212 3739 ts_tablet_manager.cc:1428] T 688cd8217fb04e4e930cb2421bfcfd9b P 5017c179ddc640fc9ebf5dea7f3957f8: Time spent starting tablet: real 0.013s user 0.003s sys 0.003s
I20250114 02:55:11.413781 3738 ts_tablet_manager.cc:1428] T 688cd8217fb04e4e930cb2421bfcfd9b P 7b7de0f9038c4ba287750ad132cb3597: Time spent starting tablet: real 0.011s user 0.006s sys 0.001s
@ 0x7fb916e6407c kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7fb916e67d32 kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
@ 0x7fb916e67cea _ZNSt3__18__invokeIRZN4kudu6master30HmsNotificationLogListenerTask4InitEvE3$_0JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
I20250114 02:55:11.473027 2977 master_service.cc:946] ReplaceTablet: received request to replace tablet 91303b0646af43409c3ad051facf590e from {username='slave'} at 127.0.0.1:57960
@ 0x7fb916e67c7a std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
I20250114 02:55:11.495599 2977 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27: Sending DeleteTablet for 3 replicas of tablet 91303b0646af43409c3ad051facf590e
I20250114 02:55:11.498657 3327 tablet_service.cc:1514] Processing DeleteTablet for tablet 91303b0646af43409c3ad051facf590e with delete_type TABLET_DATA_DELETED (replaced by tablet 5e5a4f58a4ba47e59477208eb2bd833b) from {username='slave'} at 127.0.0.1:58826
I20250114 02:55:11.499400 3471 tablet_service.cc:1514] Processing DeleteTablet for tablet 91303b0646af43409c3ad051facf590e with delete_type TABLET_DATA_DELETED (replaced by tablet 5e5a4f58a4ba47e59477208eb2bd833b) from {username='slave'} at 127.0.0.1:58610
I20250114 02:55:11.500164 3742 tablet_replica.cc:331] T 91303b0646af43409c3ad051facf590e P 7b7de0f9038c4ba287750ad132cb3597: stopping tablet replica
I20250114 02:55:11.500967 3742 raft_consensus.cc:2238] T 91303b0646af43409c3ad051facf590e P 7b7de0f9038c4ba287750ad132cb3597 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250114 02:55:11.501456 3742 raft_consensus.cc:2267] T 91303b0646af43409c3ad051facf590e P 7b7de0f9038c4ba287750ad132cb3597 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250114 02:55:11.502233 3743 tablet_replica.cc:331] T 91303b0646af43409c3ad051facf590e P 39c64942315c4716baaee0c251c05acd: stopping tablet replica
I20250114 02:55:11.503052 3743 raft_consensus.cc:2238] T 91303b0646af43409c3ad051facf590e P 39c64942315c4716baaee0c251c05acd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250114 02:55:11.503566 3743 raft_consensus.cc:2267] T 91303b0646af43409c3ad051facf590e P 39c64942315c4716baaee0c251c05acd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250114 02:55:11.505573 3613 tablet_service.cc:1514] Processing DeleteTablet for tablet 91303b0646af43409c3ad051facf590e with delete_type TABLET_DATA_DELETED (replaced by tablet 5e5a4f58a4ba47e59477208eb2bd833b) from {username='slave'} at 127.0.0.1:60674
I20250114 02:55:11.506343 3741 tablet_replica.cc:331] T 91303b0646af43409c3ad051facf590e P 5017c179ddc640fc9ebf5dea7f3957f8: stopping tablet replica
I20250114 02:55:11.506379 3742 ts_tablet_manager.cc:1905] T 91303b0646af43409c3ad051facf590e P 7b7de0f9038c4ba287750ad132cb3597: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:55:11.507161 3741 raft_consensus.cc:2238] T 91303b0646af43409c3ad051facf590e P 5017c179ddc640fc9ebf5dea7f3957f8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250114 02:55:11.507673 3741 raft_consensus.cc:2267] T 91303b0646af43409c3ad051facf590e P 5017c179ddc640fc9ebf5dea7f3957f8 [term 0 FOLLOWER]: Raft consensus is shut down!
@ 0x7fb916e67c42 std::__1::__function::__alloc_func<>::operator()() at ??:0
I20250114 02:55:11.511196 3741 ts_tablet_manager.cc:1905] T 91303b0646af43409c3ad051facf590e P 5017c179ddc640fc9ebf5dea7f3957f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:55:11.511726 3743 ts_tablet_manager.cc:1905] T 91303b0646af43409c3ad051facf590e P 39c64942315c4716baaee0c251c05acd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:55:11.515403 2977 catalog_manager.cc:6596] ReplaceTablet: tablet 91303b0646af43409c3ad051facf590e deleted and replaced by tablet 5e5a4f58a4ba47e59477208eb2bd833b
I20250114 02:55:11.515627 2979 master_service.cc:946] ReplaceTablet: received request to replace tablet 2e2bcc70bcab4dfb8e0d8835dbf92749 from {username='slave'} at 127.0.0.1:57960
I20250114 02:55:11.528045 3749 raft_consensus.cc:491] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 5017c179ddc640fc9ebf5dea7f3957f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:55:11.528729 3749 raft_consensus.cc:513] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 5017c179ddc640fc9ebf5dea7f3957f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
@ 0x7fb916e66f3e std::__1::__function::__func<>::operator()() at ??:0
@ 0x59ea05 std::__1::__function::__value_func<>::operator()() at ??:?
@ 0x59e83a std::__1::function<>::operator()() at ??:?
I20250114 02:55:11.543684 3749 leader_election.cc:290] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 5017c179ddc640fc9ebf5dea7f3957f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b7de0f9038c4ba287750ad132cb3597 (127.0.107.65:34329), 39c64942315c4716baaee0c251c05acd (127.0.107.66:33487)
I20250114 02:55:11.547699 2979 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a8f3d9567c014741a7e628bab3e18e27: Sending DeleteTablet for 3 replicas of tablet 2e2bcc70bcab4dfb8e0d8835dbf92749
I20250114 02:55:11.549892 2979 catalog_manager.cc:6596] ReplaceTablet: tablet 2e2bcc70bcab4dfb8e0d8835dbf92749 deleted and replaced by tablet 83abece59e0541f1ad0eefee1a8e7994
I20250114 02:55:11.550608 3471 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e2bcc70bcab4dfb8e0d8835dbf92749 with delete_type TABLET_DATA_DELETED (replaced by tablet 83abece59e0541f1ad0eefee1a8e7994) from {username='slave'} at 127.0.0.1:58610
I20250114 02:55:11.551446 3327 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e2bcc70bcab4dfb8e0d8835dbf92749 with delete_type TABLET_DATA_DELETED (replaced by tablet 83abece59e0541f1ad0eefee1a8e7994) from {username='slave'} at 127.0.0.1:58826
I20250114 02:55:11.571300 3741 ts_tablet_manager.cc:1918] T 91303b0646af43409c3ad051facf590e P 5017c179ddc640fc9ebf5dea7f3957f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:55:11.571851 3741 log.cc:1198] T 91303b0646af43409c3ad051facf590e P 5017c179ddc640fc9ebf5dea7f3957f8: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/91303b0646af43409c3ad051facf590e
@ 0x7fb90e1adfe7 kudu::Thread::SuperviseThread() at ??:0
@ 0x468a00 __tsan_thread_start_func at ??:?
I20250114 02:55:11.588124 3613 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e2bcc70bcab4dfb8e0d8835dbf92749 with delete_type TABLET_DATA_DELETED (replaced by tablet 83abece59e0541f1ad0eefee1a8e7994) from {username='slave'} at 127.0.0.1:60674
I20250114 02:55:11.595839 3741 ts_tablet_manager.cc:1939] T 91303b0646af43409c3ad051facf590e P 5017c179ddc640fc9ebf5dea7f3957f8: Deleting consensus metadata
I20250114 02:55:11.595988 3327 tablet_service.cc:1467] Processing CreateTablet for tablet 5e5a4f58a4ba47e59477208eb2bd833b (DEFAULT_TABLE table=default.table_bab0c0670ef34820b254580eeb9eef70 [id=89d59ce4ef604cce9b1e9a01a4aea791]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:55:11.597153 3327 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e5a4f58a4ba47e59477208eb2bd833b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:55:11.598510 2969 catalog_manager.cc:4872] TS 5017c179ddc640fc9ebf5dea7f3957f8 (127.0.107.67:45409): tablet 91303b0646af43409c3ad051facf590e (table default.table_bab0c0670ef34820b254580eeb9eef70 [id=89d59ce4ef604cce9b1e9a01a4aea791]) successfully deleted
I20250114 02:55:11.605123 3491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e2bcc70bcab4dfb8e0d8835dbf92749" candidate_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39c64942315c4716baaee0c251c05acd" is_pre_election: true
I20250114 02:55:11.606140 3491 raft_consensus.cc:2463] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 39c64942315c4716baaee0c251c05acd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5017c179ddc640fc9ebf5dea7f3957f8 in term 0.
I20250114 02:55:11.607784 3567 leader_election.cc:304] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 5017c179ddc640fc9ebf5dea7f3957f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39c64942315c4716baaee0c251c05acd, 5017c179ddc640fc9ebf5dea7f3957f8; no voters:
I20250114 02:55:11.608765 3749 raft_consensus.cc:2798] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 5017c179ddc640fc9ebf5dea7f3957f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
@ 0x7fb90f82d6db start_thread at ??:0
I20250114 02:55:11.609169 3749 raft_consensus.cc:491] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 5017c179ddc640fc9ebf5dea7f3957f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:55:11.609514 3749 raft_consensus.cc:3054] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 5017c179ddc640fc9ebf5dea7f3957f8 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:55:11.617151 3471 tablet_service.cc:1467] Processing CreateTablet for tablet 5e5a4f58a4ba47e59477208eb2bd833b (DEFAULT_TABLE table=default.table_bab0c0670ef34820b254580eeb9eef70 [id=89d59ce4ef604cce9b1e9a01a4aea791]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:55:11.618340 3471 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e5a4f58a4ba47e59477208eb2bd833b. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7fb90ac0f71f clone at ??:0
I20250114 02:55:11.627293 3749 raft_consensus.cc:513] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 5017c179ddc640fc9ebf5dea7f3957f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:11.629599 3613 tablet_service.cc:1467] Processing CreateTablet for tablet 5e5a4f58a4ba47e59477208eb2bd833b (DEFAULT_TABLE table=default.table_bab0c0670ef34820b254580eeb9eef70 [id=89d59ce4ef604cce9b1e9a01a4aea791]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:55:11.630681 3613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e5a4f58a4ba47e59477208eb2bd833b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:55:11.634199 3742 ts_tablet_manager.cc:1918] T 91303b0646af43409c3ad051facf590e P 7b7de0f9038c4ba287750ad132cb3597: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:55:11.634593 3742 log.cc:1198] T 91303b0646af43409c3ad051facf590e P 7b7de0f9038c4ba287750ad132cb3597: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/91303b0646af43409c3ad051facf590e
I20250114 02:55:11.635250 3742 ts_tablet_manager.cc:1939] T 91303b0646af43409c3ad051facf590e P 7b7de0f9038c4ba287750ad132cb3597: Deleting consensus metadata
I20250114 02:55:11.636703 3741 tablet_replica.cc:331] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 5017c179ddc640fc9ebf5dea7f3957f8: stopping tablet replica
I20250114 02:55:11.637490 3741 raft_consensus.cc:2238] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 5017c179ddc640fc9ebf5dea7f3957f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:55:11.637945 3741 raft_consensus.cc:2267] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 5017c179ddc640fc9ebf5dea7f3957f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:55:11.639184 3736 tablet_bootstrap.cc:492] T 5e5a4f58a4ba47e59477208eb2bd833b P 39c64942315c4716baaee0c251c05acd: Bootstrap starting.
I20250114 02:55:11.640131 3749 leader_election.cc:290] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 5017c179ddc640fc9ebf5dea7f3957f8 [CANDIDATE]: Term 1 election: Requested vote from peers 7b7de0f9038c4ba287750ad132cb3597 (127.0.107.65:34329), 39c64942315c4716baaee0c251c05acd (127.0.107.66:33487)
I20250114 02:55:11.640823 3491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e2bcc70bcab4dfb8e0d8835dbf92749" candidate_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39c64942315c4716baaee0c251c05acd"
I20250114 02:55:11.641400 3491 raft_consensus.cc:3054] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 39c64942315c4716baaee0c251c05acd [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:55:11.645798 3736 tablet_bootstrap.cc:654] T 5e5a4f58a4ba47e59477208eb2bd833b P 39c64942315c4716baaee0c251c05acd: Neither blocks nor log segments found. Creating new log.
I20250114 02:55:11.649758 3741 ts_tablet_manager.cc:1905] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 5017c179ddc640fc9ebf5dea7f3957f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:55:11.649858 3739 tablet_bootstrap.cc:492] T 5e5a4f58a4ba47e59477208eb2bd833b P 5017c179ddc640fc9ebf5dea7f3957f8: Bootstrap starting.
I20250114 02:55:11.652150 3491 raft_consensus.cc:2463] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 39c64942315c4716baaee0c251c05acd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5017c179ddc640fc9ebf5dea7f3957f8 in term 1.
I20250114 02:55:11.652604 3743 ts_tablet_manager.cc:1918] T 91303b0646af43409c3ad051facf590e P 39c64942315c4716baaee0c251c05acd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:55:11.652980 3743 log.cc:1198] T 91303b0646af43409c3ad051facf590e P 39c64942315c4716baaee0c251c05acd: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/91303b0646af43409c3ad051facf590e
I20250114 02:55:11.653628 3743 ts_tablet_manager.cc:1939] T 91303b0646af43409c3ad051facf590e P 39c64942315c4716baaee0c251c05acd: Deleting consensus metadata
I20250114 02:55:11.656067 3567 leader_election.cc:304] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 5017c179ddc640fc9ebf5dea7f3957f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39c64942315c4716baaee0c251c05acd, 5017c179ddc640fc9ebf5dea7f3957f8; no voters:
W20250114 02:55:11.656872 3567 raft_consensus.cc:2681] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 5017c179ddc640fc9ebf5dea7f3957f8: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250114 02:55:11.657979 3743 tablet_replica.cc:331] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 39c64942315c4716baaee0c251c05acd: stopping tablet replica
I20250114 02:55:11.658665 3743 raft_consensus.cc:2238] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 39c64942315c4716baaee0c251c05acd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:55:11.659111 3739 tablet_bootstrap.cc:654] T 5e5a4f58a4ba47e59477208eb2bd833b P 5017c179ddc640fc9ebf5dea7f3957f8: Neither blocks nor log segments found. Creating new log.
I20250114 02:55:11.659121 3743 raft_consensus.cc:2267] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 39c64942315c4716baaee0c251c05acd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:55:11.663112 3736 tablet_bootstrap.cc:492] T 5e5a4f58a4ba47e59477208eb2bd833b P 39c64942315c4716baaee0c251c05acd: No bootstrap required, opened a new log
I20250114 02:55:11.663544 3736 ts_tablet_manager.cc:1397] T 5e5a4f58a4ba47e59477208eb2bd833b P 39c64942315c4716baaee0c251c05acd: Time spent bootstrapping tablet: real 0.025s user 0.011s sys 0.003s
I20250114 02:55:11.664965 3743 ts_tablet_manager.cc:1905] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 39c64942315c4716baaee0c251c05acd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:55:11.665848 3738 tablet_bootstrap.cc:492] T 5e5a4f58a4ba47e59477208eb2bd833b P 7b7de0f9038c4ba287750ad132cb3597: Bootstrap starting.
I20250114 02:55:11.665999 3736 raft_consensus.cc:357] T 5e5a4f58a4ba47e59477208eb2bd833b P 39c64942315c4716baaee0c251c05acd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:11.666714 3736 raft_consensus.cc:383] T 5e5a4f58a4ba47e59477208eb2bd833b P 39c64942315c4716baaee0c251c05acd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:55:11.666980 3736 raft_consensus.cc:738] T 5e5a4f58a4ba47e59477208eb2bd833b P 39c64942315c4716baaee0c251c05acd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39c64942315c4716baaee0c251c05acd, State: Initialized, Role: FOLLOWER
I20250114 02:55:11.667609 3736 consensus_queue.cc:260] T 5e5a4f58a4ba47e59477208eb2bd833b P 39c64942315c4716baaee0c251c05acd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:11.669886 3736 ts_tablet_manager.cc:1428] T 5e5a4f58a4ba47e59477208eb2bd833b P 39c64942315c4716baaee0c251c05acd: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s
I20250114 02:55:11.671882 3738 tablet_bootstrap.cc:654] T 5e5a4f58a4ba47e59477208eb2bd833b P 7b7de0f9038c4ba287750ad132cb3597: Neither blocks nor log segments found. Creating new log.
I20250114 02:55:11.675824 3748 raft_consensus.cc:491] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 7b7de0f9038c4ba287750ad132cb3597 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:55:11.676424 3748 raft_consensus.cc:513] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 7b7de0f9038c4ba287750ad132cb3597 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:11.679780 3347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e2bcc70bcab4dfb8e0d8835dbf92749" candidate_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b7de0f9038c4ba287750ad132cb3597" is_pre_election: true
I20250114 02:55:11.680493 3748 leader_election.cc:290] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 7b7de0f9038c4ba287750ad132cb3597 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39c64942315c4716baaee0c251c05acd (127.0.107.66:33487), 5017c179ddc640fc9ebf5dea7f3957f8 (127.0.107.67:45409)
I20250114 02:55:11.700001 3742 tablet_replica.cc:331] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 7b7de0f9038c4ba287750ad132cb3597: stopping tablet replica
I20250114 02:55:11.700275 3346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e2bcc70bcab4dfb8e0d8835dbf92749" candidate_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b7de0f9038c4ba287750ad132cb3597"
I20250114 02:55:11.701004 3742 raft_consensus.cc:2238] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 7b7de0f9038c4ba287750ad132cb3597 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250114 02:55:11.701521 3742 raft_consensus.cc:2267] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 7b7de0f9038c4ba287750ad132cb3597 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250114 02:55:11.704509 3749 raft_consensus.cc:491] T c174bcbc5d51402b8ae332b87ffa2139 P 5017c179ddc640fc9ebf5dea7f3957f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:55:11.704962 3749 raft_consensus.cc:513] T c174bcbc5d51402b8ae332b87ffa2139 P 5017c179ddc640fc9ebf5dea7f3957f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:11.705960 3741 ts_tablet_manager.cc:1918] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 5017c179ddc640fc9ebf5dea7f3957f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:55:11.706357 3741 log.cc:1198] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 5017c179ddc640fc9ebf5dea7f3957f8: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/2e2bcc70bcab4dfb8e0d8835dbf92749
I20250114 02:55:11.706815 3749 leader_election.cc:290] T c174bcbc5d51402b8ae332b87ffa2139 P 5017c179ddc640fc9ebf5dea7f3957f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b7de0f9038c4ba287750ad132cb3597 (127.0.107.65:34329), 39c64942315c4716baaee0c251c05acd (127.0.107.66:33487)
I20250114 02:55:11.707007 3741 ts_tablet_manager.cc:1939] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 5017c179ddc640fc9ebf5dea7f3957f8: Deleting consensus metadata
W20250114 02:55:11.710217 3567 leader_election.cc:343] T c174bcbc5d51402b8ae332b87ffa2139 P 5017c179ddc640fc9ebf5dea7f3957f8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39c64942315c4716baaee0c251c05acd (127.0.107.66:33487): Not found: Tablet not found: c174bcbc5d51402b8ae332b87ffa2139
I20250114 02:55:11.718233 3742 ts_tablet_manager.cc:1905] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 7b7de0f9038c4ba287750ad132cb3597: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:55:11.718413 3347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c174bcbc5d51402b8ae332b87ffa2139" candidate_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b7de0f9038c4ba287750ad132cb3597" is_pre_election: true
I20250114 02:55:11.724463 3491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e2bcc70bcab4dfb8e0d8835dbf92749" candidate_uuid: "7b7de0f9038c4ba287750ad132cb3597" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39c64942315c4716baaee0c251c05acd" is_pre_election: true
I20250114 02:55:11.724738 3347 raft_consensus.cc:2463] T c174bcbc5d51402b8ae332b87ffa2139 P 7b7de0f9038c4ba287750ad132cb3597 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5017c179ddc640fc9ebf5dea7f3957f8 in term 0.
I20250114 02:55:11.726217 3566 leader_election.cc:304] T c174bcbc5d51402b8ae332b87ffa2139 P 5017c179ddc640fc9ebf5dea7f3957f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5017c179ddc640fc9ebf5dea7f3957f8, 7b7de0f9038c4ba287750ad132cb3597; no voters: 39c64942315c4716baaee0c251c05acd
W20250114 02:55:11.727523 3566 leader_election.cc:343] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 5017c179ddc640fc9ebf5dea7f3957f8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7b7de0f9038c4ba287750ad132cb3597 (127.0.107.65:34329): Illegal state: must be running to vote when last-logged opid is not known
I20250114 02:55:11.728384 3749 raft_consensus.cc:2798] T c174bcbc5d51402b8ae332b87ffa2139 P 5017c179ddc640fc9ebf5dea7f3957f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:55:11.728724 3749 raft_consensus.cc:491] T c174bcbc5d51402b8ae332b87ffa2139 P 5017c179ddc640fc9ebf5dea7f3957f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:55:11.729069 3749 raft_consensus.cc:3054] T c174bcbc5d51402b8ae332b87ffa2139 P 5017c179ddc640fc9ebf5dea7f3957f8 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:55:11.731961 3738 tablet_bootstrap.cc:492] T 5e5a4f58a4ba47e59477208eb2bd833b P 7b7de0f9038c4ba287750ad132cb3597: No bootstrap required, opened a new log
I20250114 02:55:11.732426 3738 ts_tablet_manager.cc:1397] T 5e5a4f58a4ba47e59477208eb2bd833b P 7b7de0f9038c4ba287750ad132cb3597: Time spent bootstrapping tablet: real 0.067s user 0.003s sys 0.008s
W20250114 02:55:11.734018 3281 leader_election.cc:343] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 7b7de0f9038c4ba287750ad132cb3597 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39c64942315c4716baaee0c251c05acd (127.0.107.66:33487): Illegal state: Tablet not RUNNING: STOPPED
I20250114 02:55:11.734892 3738 raft_consensus.cc:357] T 5e5a4f58a4ba47e59477208eb2bd833b P 7b7de0f9038c4ba287750ad132cb3597 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:11.735607 3738 raft_consensus.cc:383] T 5e5a4f58a4ba47e59477208eb2bd833b P 7b7de0f9038c4ba287750ad132cb3597 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:55:11.735872 3738 raft_consensus.cc:738] T 5e5a4f58a4ba47e59477208eb2bd833b P 7b7de0f9038c4ba287750ad132cb3597 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b7de0f9038c4ba287750ad132cb3597, State: Initialized, Role: FOLLOWER
I20250114 02:55:11.738858 3749 raft_consensus.cc:513] T c174bcbc5d51402b8ae332b87ffa2139 P 5017c179ddc640fc9ebf5dea7f3957f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:11.741583 3346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c174bcbc5d51402b8ae332b87ffa2139" candidate_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b7de0f9038c4ba287750ad132cb3597"
I20250114 02:55:11.742179 3346 raft_consensus.cc:3054] T c174bcbc5d51402b8ae332b87ffa2139 P 7b7de0f9038c4ba287750ad132cb3597 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:55:11.749898 3346 raft_consensus.cc:2463] T c174bcbc5d51402b8ae332b87ffa2139 P 7b7de0f9038c4ba287750ad132cb3597 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5017c179ddc640fc9ebf5dea7f3957f8 in term 1.
I20250114 02:55:11.750551 3742 ts_tablet_manager.cc:1918] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 7b7de0f9038c4ba287750ad132cb3597: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:55:11.750901 3742 log.cc:1198] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 7b7de0f9038c4ba287750ad132cb3597: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/2e2bcc70bcab4dfb8e0d8835dbf92749
I20250114 02:55:11.751188 3566 leader_election.cc:304] T c174bcbc5d51402b8ae332b87ffa2139 P 5017c179ddc640fc9ebf5dea7f3957f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5017c179ddc640fc9ebf5dea7f3957f8, 7b7de0f9038c4ba287750ad132cb3597; no voters:
I20250114 02:55:11.751546 3742 ts_tablet_manager.cc:1939] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 7b7de0f9038c4ba287750ad132cb3597: Deleting consensus metadata
W20250114 02:55:11.756199 3567 leader_election.cc:343] T c174bcbc5d51402b8ae332b87ffa2139 P 5017c179ddc640fc9ebf5dea7f3957f8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 39c64942315c4716baaee0c251c05acd (127.0.107.66:33487): Not found: Tablet not found: c174bcbc5d51402b8ae332b87ffa2139
I20250114 02:55:11.757750 3749 leader_election.cc:290] T c174bcbc5d51402b8ae332b87ffa2139 P 5017c179ddc640fc9ebf5dea7f3957f8 [CANDIDATE]: Term 1 election: Requested vote from peers 7b7de0f9038c4ba287750ad132cb3597 (127.0.107.65:34329), 39c64942315c4716baaee0c251c05acd (127.0.107.66:33487)
I20250114 02:55:11.758262 3749 raft_consensus.cc:2798] T c174bcbc5d51402b8ae332b87ffa2139 P 5017c179ddc640fc9ebf5dea7f3957f8 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:55:11.758694 3749 raft_consensus.cc:695] T c174bcbc5d51402b8ae332b87ffa2139 P 5017c179ddc640fc9ebf5dea7f3957f8 [term 1 LEADER]: Becoming Leader. State: Replica: 5017c179ddc640fc9ebf5dea7f3957f8, State: Running, Role: LEADER
I20250114 02:55:11.763520 3739 tablet_bootstrap.cc:492] T 5e5a4f58a4ba47e59477208eb2bd833b P 5017c179ddc640fc9ebf5dea7f3957f8: No bootstrap required, opened a new log
I20250114 02:55:11.764067 3748 raft_consensus.cc:491] T 688cd8217fb04e4e930cb2421bfcfd9b P 7b7de0f9038c4ba287750ad132cb3597 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:55:11.764532 3748 raft_consensus.cc:513] T 688cd8217fb04e4e930cb2421bfcfd9b P 7b7de0f9038c4ba287750ad132cb3597 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:11.766311 3748 leader_election.cc:290] T 688cd8217fb04e4e930cb2421bfcfd9b P 7b7de0f9038c4ba287750ad132cb3597 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39c64942315c4716baaee0c251c05acd (127.0.107.66:33487), 5017c179ddc640fc9ebf5dea7f3957f8 (127.0.107.67:45409)
I20250114 02:55:11.768656 3491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "688cd8217fb04e4e930cb2421bfcfd9b" candidate_uuid: "7b7de0f9038c4ba287750ad132cb3597" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39c64942315c4716baaee0c251c05acd" is_pre_election: true
I20250114 02:55:11.769281 3491 raft_consensus.cc:2463] T 688cd8217fb04e4e930cb2421bfcfd9b P 39c64942315c4716baaee0c251c05acd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b7de0f9038c4ba287750ad132cb3597 in term 0.
I20250114 02:55:11.769758 3738 consensus_queue.cc:260] T 5e5a4f58a4ba47e59477208eb2bd833b P 7b7de0f9038c4ba287750ad132cb3597 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:11.770546 3281 leader_election.cc:304] T 688cd8217fb04e4e930cb2421bfcfd9b P 7b7de0f9038c4ba287750ad132cb3597 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39c64942315c4716baaee0c251c05acd, 7b7de0f9038c4ba287750ad132cb3597; no voters:
I20250114 02:55:11.771335 3739 ts_tablet_manager.cc:1397] T 5e5a4f58a4ba47e59477208eb2bd833b P 5017c179ddc640fc9ebf5dea7f3957f8: Time spent bootstrapping tablet: real 0.122s user 0.009s sys 0.017s
I20250114 02:55:11.774019 3739 raft_consensus.cc:357] T 5e5a4f58a4ba47e59477208eb2bd833b P 5017c179ddc640fc9ebf5dea7f3957f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:11.774751 3739 raft_consensus.cc:383] T 5e5a4f58a4ba47e59477208eb2bd833b P 5017c179ddc640fc9ebf5dea7f3957f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:55:11.775038 3739 raft_consensus.cc:738] T 5e5a4f58a4ba47e59477208eb2bd833b P 5017c179ddc640fc9ebf5dea7f3957f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5017c179ddc640fc9ebf5dea7f3957f8, State: Initialized, Role: FOLLOWER
I20250114 02:55:11.775705 3739 consensus_queue.cc:260] T 5e5a4f58a4ba47e59477208eb2bd833b P 5017c179ddc640fc9ebf5dea7f3957f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:11.778873 3739 ts_tablet_manager.cc:1428] T 5e5a4f58a4ba47e59477208eb2bd833b P 5017c179ddc640fc9ebf5dea7f3957f8: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250114 02:55:11.782063 3748 raft_consensus.cc:2798] T 688cd8217fb04e4e930cb2421bfcfd9b P 7b7de0f9038c4ba287750ad132cb3597 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:55:11.782411 3748 raft_consensus.cc:491] T 688cd8217fb04e4e930cb2421bfcfd9b P 7b7de0f9038c4ba287750ad132cb3597 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:55:11.782702 3748 raft_consensus.cc:3054] T 688cd8217fb04e4e930cb2421bfcfd9b P 7b7de0f9038c4ba287750ad132cb3597 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:55:11.793013 3748 raft_consensus.cc:513] T 688cd8217fb04e4e930cb2421bfcfd9b P 7b7de0f9038c4ba287750ad132cb3597 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:11.795501 3748 leader_election.cc:290] T 688cd8217fb04e4e930cb2421bfcfd9b P 7b7de0f9038c4ba287750ad132cb3597 [CANDIDATE]: Term 1 election: Requested vote from peers 39c64942315c4716baaee0c251c05acd (127.0.107.66:33487), 5017c179ddc640fc9ebf5dea7f3957f8 (127.0.107.67:45409)
I20250114 02:55:11.761730 3749 consensus_queue.cc:237] T c174bcbc5d51402b8ae332b87ffa2139 P 5017c179ddc640fc9ebf5dea7f3957f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:11.802337 3738 ts_tablet_manager.cc:1428] T 5e5a4f58a4ba47e59477208eb2bd833b P 7b7de0f9038c4ba287750ad132cb3597: Time spent starting tablet: real 0.070s user 0.003s sys 0.002s
I20250114 02:55:11.803076 3491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "688cd8217fb04e4e930cb2421bfcfd9b" candidate_uuid: "7b7de0f9038c4ba287750ad132cb3597" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39c64942315c4716baaee0c251c05acd"
I20250114 02:55:11.803646 3491 raft_consensus.cc:3054] T 688cd8217fb04e4e930cb2421bfcfd9b P 39c64942315c4716baaee0c251c05acd [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:55:11.814924 3491 raft_consensus.cc:2463] T 688cd8217fb04e4e930cb2421bfcfd9b P 39c64942315c4716baaee0c251c05acd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b7de0f9038c4ba287750ad132cb3597 in term 1.
I20250114 02:55:11.818414 3633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "688cd8217fb04e4e930cb2421bfcfd9b" candidate_uuid: "7b7de0f9038c4ba287750ad132cb3597" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" is_pre_election: true
I20250114 02:55:11.819144 3633 raft_consensus.cc:2463] T 688cd8217fb04e4e930cb2421bfcfd9b P 5017c179ddc640fc9ebf5dea7f3957f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b7de0f9038c4ba287750ad132cb3597 in term 0.
I20250114 02:55:11.820515 3633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "688cd8217fb04e4e930cb2421bfcfd9b" candidate_uuid: "7b7de0f9038c4ba287750ad132cb3597" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5017c179ddc640fc9ebf5dea7f3957f8"
W20250114 02:55:11.820714 3280 leader_election.cc:343] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 7b7de0f9038c4ba287750ad132cb3597 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5017c179ddc640fc9ebf5dea7f3957f8 (127.0.107.67:45409): Not found: Tablet not found: 2e2bcc70bcab4dfb8e0d8835dbf92749
I20250114 02:55:11.821070 3633 raft_consensus.cc:3054] T 688cd8217fb04e4e930cb2421bfcfd9b P 5017c179ddc640fc9ebf5dea7f3957f8 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:55:11.821130 3280 leader_election.cc:304] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 7b7de0f9038c4ba287750ad132cb3597 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b7de0f9038c4ba287750ad132cb3597; no voters: 39c64942315c4716baaee0c251c05acd, 5017c179ddc640fc9ebf5dea7f3957f8
W20250114 02:55:11.822657 3280 raft_consensus.cc:2681] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 7b7de0f9038c4ba287750ad132cb3597: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250114 02:55:11.824741 3743 ts_tablet_manager.cc:1918] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 39c64942315c4716baaee0c251c05acd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:55:11.825232 3743 log.cc:1198] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 39c64942315c4716baaee0c251c05acd: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/2e2bcc70bcab4dfb8e0d8835dbf92749
I20250114 02:55:11.825402 3281 leader_election.cc:304] T 688cd8217fb04e4e930cb2421bfcfd9b P 7b7de0f9038c4ba287750ad132cb3597 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39c64942315c4716baaee0c251c05acd, 7b7de0f9038c4ba287750ad132cb3597; no voters:
I20250114 02:55:11.826076 3743 ts_tablet_manager.cc:1939] T 2e2bcc70bcab4dfb8e0d8835dbf92749 P 39c64942315c4716baaee0c251c05acd: Deleting consensus metadata
I20250114 02:55:11.826349 3784 raft_consensus.cc:2798] T 688cd8217fb04e4e930cb2421bfcfd9b P 7b7de0f9038c4ba287750ad132cb3597 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:55:11.826833 3784 raft_consensus.cc:695] T 688cd8217fb04e4e930cb2421bfcfd9b P 7b7de0f9038c4ba287750ad132cb3597 [term 1 LEADER]: Becoming Leader. State: Replica: 7b7de0f9038c4ba287750ad132cb3597, State: Running, Role: LEADER
I20250114 02:55:11.829099 3784 consensus_queue.cc:237] T 688cd8217fb04e4e930cb2421bfcfd9b P 7b7de0f9038c4ba287750ad132cb3597 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:11.836167 3633 raft_consensus.cc:2463] T 688cd8217fb04e4e930cb2421bfcfd9b P 5017c179ddc640fc9ebf5dea7f3957f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b7de0f9038c4ba287750ad132cb3597 in term 1.
W20250114 02:55:11.911844 3751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 02:55:11.912710 3751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 02:55:11.913551 3751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 02:55:11.973827 3751 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20250114 02:55:11.974221 3751 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20250114 02:55:11.974570 3751 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250114 02:55:11.974895 3751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 02:55:11.975222 3751 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250114 02:55:11.975528 3751 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20250114 02:55:12.023916 3426 connection.cc:537] server connection from 127.0.0.1:58610 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250114 02:55:12.023973 3690 connection.cc:537] client connection to 127.0.107.126:40235 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250114 02:55:12.024781 3567 connection.cc:537] server connection from 127.0.0.1:60674 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250114 02:55:12.024892 3568 connection.cc:537] client connection to 127.0.107.126:40235 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250114 02:55:12.026083 3690 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.0.107.126:40235)
W20250114 02:55:12.026129 3282 connection.cc:537] client connection to 127.0.107.126:40235 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250114 02:55:12.026816 3690 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.107.126:40235)
W20250114 02:55:12.026895 3280 connection.cc:537] server connection from 127.0.0.1:58826 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250114 02:55:12.028102 3425 connection.cc:537] client connection to 127.0.107.126:40235 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250114 02:55:12.035182 3690 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.0.107.126:40235)
W20250114 02:55:12.036072 3690 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.107.126:40235)
I20250114 02:55:12.053002 3751 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.107.84:38241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:33943
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.107.126:40235,127.0.107.125:38653,127.0.107.124:44431
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.107.124:44431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.107.124
--webserver_port=36967
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type FASTDEBUG
built by None at 14 Jan 2025 02:43:23 UTC on a633016ff5c9
build id 4168
TSAN enabled
I20250114 02:55:12.054364 3751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 02:55:12.056049 3751 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 02:55:12.074239 3799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 02:55:12.092192 3804 raft_consensus.cc:491] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a8f3d9567c014741a7e628bab3e18e27)
I20250114 02:55:12.092692 3804 raft_consensus.cc:513] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:55:12.112998 3804 leader_election.cc:290] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235), f9a6e2eab6e541eca12ee79c1d3004e8 (127.0.107.124:44431)
W20250114 02:55:12.115061 3046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.107.124:44431: connect: Connection refused (error 111) [suppressed 6 similar messages]
W20250114 02:55:12.124068 3046 leader_election.cc:336] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f9a6e2eab6e541eca12ee79c1d3004e8 (127.0.107.124:44431): Network error: Client connection negotiation failed: client connection to 127.0.107.124:44431: connect: Connection refused (error 111)
W20250114 02:55:12.125728 3045 leader_election.cc:336] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235): Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111)
I20250114 02:55:12.126106 3045 leader_election.cc:304] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bfb2e2fecfb94af8bf892c6f774c2d89; no voters: a8f3d9567c014741a7e628bab3e18e27, f9a6e2eab6e541eca12ee79c1d3004e8
I20250114 02:55:12.126796 3804 raft_consensus.cc:2743] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 02:55:12.235379 3809 raft_consensus.cc:491] T 5e5a4f58a4ba47e59477208eb2bd833b P 39c64942315c4716baaee0c251c05acd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:55:12.240686 3809 raft_consensus.cc:513] T 5e5a4f58a4ba47e59477208eb2bd833b P 39c64942315c4716baaee0c251c05acd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:12.301007 3809 leader_election.cc:290] T 5e5a4f58a4ba47e59477208eb2bd833b P 39c64942315c4716baaee0c251c05acd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b7de0f9038c4ba287750ad132cb3597 (127.0.107.65:34329), 5017c179ddc640fc9ebf5dea7f3957f8 (127.0.107.67:45409)
I20250114 02:55:12.322021 3784 raft_consensus.cc:491] T 5e5a4f58a4ba47e59477208eb2bd833b P 7b7de0f9038c4ba287750ad132cb3597 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:55:12.322743 3784 raft_consensus.cc:513] T 5e5a4f58a4ba47e59477208eb2bd833b P 7b7de0f9038c4ba287750ad132cb3597 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:12.326663 3491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e5a4f58a4ba47e59477208eb2bd833b" candidate_uuid: "7b7de0f9038c4ba287750ad132cb3597" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39c64942315c4716baaee0c251c05acd" is_pre_election: true
I20250114 02:55:12.327685 3491 raft_consensus.cc:2463] T 5e5a4f58a4ba47e59477208eb2bd833b P 39c64942315c4716baaee0c251c05acd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b7de0f9038c4ba287750ad132cb3597 in term 0.
I20250114 02:55:12.330317 3281 leader_election.cc:304] T 5e5a4f58a4ba47e59477208eb2bd833b P 7b7de0f9038c4ba287750ad132cb3597 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39c64942315c4716baaee0c251c05acd, 7b7de0f9038c4ba287750ad132cb3597; no voters:
I20250114 02:55:12.333197 3784 leader_election.cc:290] T 5e5a4f58a4ba47e59477208eb2bd833b P 7b7de0f9038c4ba287750ad132cb3597 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39c64942315c4716baaee0c251c05acd (127.0.107.66:33487), 5017c179ddc640fc9ebf5dea7f3957f8 (127.0.107.67:45409)
I20250114 02:55:12.334077 3784 raft_consensus.cc:2798] T 5e5a4f58a4ba47e59477208eb2bd833b P 7b7de0f9038c4ba287750ad132cb3597 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:55:12.334599 3784 raft_consensus.cc:491] T 5e5a4f58a4ba47e59477208eb2bd833b P 7b7de0f9038c4ba287750ad132cb3597 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:55:12.335206 3784 raft_consensus.cc:3054] T 5e5a4f58a4ba47e59477208eb2bd833b P 7b7de0f9038c4ba287750ad132cb3597 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:55:12.345001 3784 raft_consensus.cc:513] T 5e5a4f58a4ba47e59477208eb2bd833b P 7b7de0f9038c4ba287750ad132cb3597 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:12.352406 3491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e5a4f58a4ba47e59477208eb2bd833b" candidate_uuid: "7b7de0f9038c4ba287750ad132cb3597" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39c64942315c4716baaee0c251c05acd"
I20250114 02:55:12.353286 3491 raft_consensus.cc:3054] T 5e5a4f58a4ba47e59477208eb2bd833b P 39c64942315c4716baaee0c251c05acd [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:55:12.363315 3491 raft_consensus.cc:2463] T 5e5a4f58a4ba47e59477208eb2bd833b P 39c64942315c4716baaee0c251c05acd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b7de0f9038c4ba287750ad132cb3597 in term 1.
I20250114 02:55:12.364996 3281 leader_election.cc:304] T 5e5a4f58a4ba47e59477208eb2bd833b P 7b7de0f9038c4ba287750ad132cb3597 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39c64942315c4716baaee0c251c05acd, 7b7de0f9038c4ba287750ad132cb3597; no voters:
I20250114 02:55:12.366078 3812 raft_consensus.cc:2798] T 5e5a4f58a4ba47e59477208eb2bd833b P 7b7de0f9038c4ba287750ad132cb3597 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:55:12.368873 3784 leader_election.cc:290] T 5e5a4f58a4ba47e59477208eb2bd833b P 7b7de0f9038c4ba287750ad132cb3597 [CANDIDATE]: Term 1 election: Requested vote from peers 39c64942315c4716baaee0c251c05acd (127.0.107.66:33487), 5017c179ddc640fc9ebf5dea7f3957f8 (127.0.107.67:45409)
I20250114 02:55:12.384528 3812 raft_consensus.cc:695] T 5e5a4f58a4ba47e59477208eb2bd833b P 7b7de0f9038c4ba287750ad132cb3597 [term 1 LEADER]: Becoming Leader. State: Replica: 7b7de0f9038c4ba287750ad132cb3597, State: Running, Role: LEADER
I20250114 02:55:12.385627 3812 consensus_queue.cc:237] T 5e5a4f58a4ba47e59477208eb2bd833b P 7b7de0f9038c4ba287750ad132cb3597 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:12.410851 3346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e5a4f58a4ba47e59477208eb2bd833b" candidate_uuid: "39c64942315c4716baaee0c251c05acd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b7de0f9038c4ba287750ad132cb3597" is_pre_election: true
I20250114 02:55:12.415513 3813 raft_consensus.cc:491] T 5e5a4f58a4ba47e59477208eb2bd833b P 5017c179ddc640fc9ebf5dea7f3957f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:55:12.416117 3813 raft_consensus.cc:513] T 5e5a4f58a4ba47e59477208eb2bd833b P 5017c179ddc640fc9ebf5dea7f3957f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:12.418817 3633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e5a4f58a4ba47e59477208eb2bd833b" candidate_uuid: "7b7de0f9038c4ba287750ad132cb3597" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" is_pre_election: true
I20250114 02:55:12.419654 3633 raft_consensus.cc:2463] T 5e5a4f58a4ba47e59477208eb2bd833b P 5017c179ddc640fc9ebf5dea7f3957f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b7de0f9038c4ba287750ad132cb3597 in term 0.
I20250114 02:55:12.420991 3632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e5a4f58a4ba47e59477208eb2bd833b" candidate_uuid: "7b7de0f9038c4ba287750ad132cb3597" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5017c179ddc640fc9ebf5dea7f3957f8"
I20250114 02:55:12.421836 3632 raft_consensus.cc:3054] T 5e5a4f58a4ba47e59477208eb2bd833b P 5017c179ddc640fc9ebf5dea7f3957f8 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:55:12.423233 3633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e5a4f58a4ba47e59477208eb2bd833b" candidate_uuid: "39c64942315c4716baaee0c251c05acd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" is_pre_election: true
I20250114 02:55:12.425177 3423 leader_election.cc:304] T 5e5a4f58a4ba47e59477208eb2bd833b P 39c64942315c4716baaee0c251c05acd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39c64942315c4716baaee0c251c05acd; no voters: 5017c179ddc640fc9ebf5dea7f3957f8, 7b7de0f9038c4ba287750ad132cb3597
I20250114 02:55:12.426813 3809 raft_consensus.cc:2743] T 5e5a4f58a4ba47e59477208eb2bd833b P 39c64942315c4716baaee0c251c05acd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:55:12.431375 3632 raft_consensus.cc:2463] T 5e5a4f58a4ba47e59477208eb2bd833b P 5017c179ddc640fc9ebf5dea7f3957f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b7de0f9038c4ba287750ad132cb3597 in term 1.
I20250114 02:55:12.447374 3346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e5a4f58a4ba47e59477208eb2bd833b" candidate_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b7de0f9038c4ba287750ad132cb3597" is_pre_election: true
I20250114 02:55:12.451418 3491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e5a4f58a4ba47e59477208eb2bd833b" candidate_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39c64942315c4716baaee0c251c05acd" is_pre_election: true
I20250114 02:55:12.452540 3491 raft_consensus.cc:2388] T 5e5a4f58a4ba47e59477208eb2bd833b P 39c64942315c4716baaee0c251c05acd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5017c179ddc640fc9ebf5dea7f3957f8 in current term 1: Already voted for candidate 7b7de0f9038c4ba287750ad132cb3597 in this term.
I20250114 02:55:12.454162 3567 leader_election.cc:304] T 5e5a4f58a4ba47e59477208eb2bd833b P 5017c179ddc640fc9ebf5dea7f3957f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5017c179ddc640fc9ebf5dea7f3957f8; no voters: 39c64942315c4716baaee0c251c05acd, 7b7de0f9038c4ba287750ad132cb3597
I20250114 02:55:12.460544 3813 leader_election.cc:290] T 5e5a4f58a4ba47e59477208eb2bd833b P 5017c179ddc640fc9ebf5dea7f3957f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b7de0f9038c4ba287750ad132cb3597 (127.0.107.65:34329), 39c64942315c4716baaee0c251c05acd (127.0.107.66:33487)
I20250114 02:55:12.461251 3813 raft_consensus.cc:2743] T 5e5a4f58a4ba47e59477208eb2bd833b P 5017c179ddc640fc9ebf5dea7f3957f8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 02:55:12.598953 3812 consensus_queue.cc:1035] T 688cd8217fb04e4e930cb2421bfcfd9b P 7b7de0f9038c4ba287750ad132cb3597 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:55:12.717969 3784 consensus_queue.cc:1035] T 688cd8217fb04e4e930cb2421bfcfd9b P 7b7de0f9038c4ba287750ad132cb3597 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20250114 02:55:12.771632 3823 raft_consensus.cc:491] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a8f3d9567c014741a7e628bab3e18e27)
W20250114 02:55:12.773202 3567 consensus_peers.cc:487] T c174bcbc5d51402b8ae332b87ffa2139 P 5017c179ddc640fc9ebf5dea7f3957f8 -> Peer 39c64942315c4716baaee0c251c05acd (127.0.107.66:33487): Couldn't send request to peer 39c64942315c4716baaee0c251c05acd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c174bcbc5d51402b8ae332b87ffa2139. This is attempt 1: this message will repeat every 5th retry.
I20250114 02:55:12.772253 3823 raft_consensus.cc:513] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:55:12.783409 3823 leader_election.cc:290] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235), f9a6e2eab6e541eca12ee79c1d3004e8 (127.0.107.124:44431)
I20250114 02:55:12.789088 3816 consensus_queue.cc:1035] T c174bcbc5d51402b8ae332b87ffa2139 P 5017c179ddc640fc9ebf5dea7f3957f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250114 02:55:12.815585 3045 leader_election.cc:336] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235): Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111)
W20250114 02:55:12.840543 3046 leader_election.cc:336] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f9a6e2eab6e541eca12ee79c1d3004e8 (127.0.107.124:44431): Network error: Client connection negotiation failed: client connection to 127.0.107.124:44431: connect: Connection refused (error 111)
I20250114 02:55:12.840984 3046 leader_election.cc:304] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bfb2e2fecfb94af8bf892c6f774c2d89; no voters: a8f3d9567c014741a7e628bab3e18e27, f9a6e2eab6e541eca12ee79c1d3004e8
I20250114 02:55:12.841713 3823 raft_consensus.cc:2743] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 02:55:13.318766 3784 consensus_queue.cc:1035] T 5e5a4f58a4ba47e59477208eb2bd833b P 7b7de0f9038c4ba287750ad132cb3597 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250114 02:55:13.409529 3784 consensus_queue.cc:1035] T 5e5a4f58a4ba47e59477208eb2bd833b P 7b7de0f9038c4ba287750ad132cb3597 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:55:13.640455 3843 raft_consensus.cc:491] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a8f3d9567c014741a7e628bab3e18e27)
I20250114 02:55:13.641209 3843 raft_consensus.cc:513] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:55:13.659843 3843 leader_election.cc:290] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235), f9a6e2eab6e541eca12ee79c1d3004e8 (127.0.107.124:44431)
W20250114 02:55:13.687399 3046 leader_election.cc:336] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f9a6e2eab6e541eca12ee79c1d3004e8 (127.0.107.124:44431): Network error: Client connection negotiation failed: client connection to 127.0.107.124:44431: connect: Connection refused (error 111)
W20250114 02:55:13.688192 3045 leader_election.cc:336] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235): Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111)
I20250114 02:55:13.688832 3045 leader_election.cc:304] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bfb2e2fecfb94af8bf892c6f774c2d89; no voters: a8f3d9567c014741a7e628bab3e18e27, f9a6e2eab6e541eca12ee79c1d3004e8
I20250114 02:55:13.689846 3843 raft_consensus.cc:2743] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250114 02:55:13.474762 3798 debug-util.cc:398] Leaking SignalData structure 0x7b080001ca00 after lost signal to thread 3751
W20250114 02:55:14.088733 3798 kernel_stack_watchdog.cc:198] Thread 3751 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:636 for 397ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250114 02:55:12.076093 3800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 02:55:14.090049 3751 thread.cc:635] OpenStack (cloud detector) Time spent creating pthread: real 2.015s user 0.617s sys 1.254s
W20250114 02:55:14.091044 3751 thread.cc:601] OpenStack (cloud detector) Time spent starting thread: real 2.016s user 0.617s sys 1.254s
W20250114 02:55:14.092937 3802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 02:55:14.095712 3801 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 2013 milliseconds
I20250114 02:55:14.095736 3751 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250114 02:55:14.097128 3751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 02:55:14.100816 3751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 02:55:14.102408 3751 hybrid_clock.cc:648] HybridClock initialized: now 1736823314102357 us; error 49 us; skew 500 ppm
I20250114 02:55:14.103194 3751 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250114 02:55:14.113543 3751 webserver.cc:458] Webserver started at http://127.0.107.124:36967/ using document root <none> and password file <none>
I20250114 02:55:14.114511 3751 fs_manager.cc:362] Metadata directory not provided
I20250114 02:55:14.114758 3751 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 02:55:14.122857 3751 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.001s sys 0.005s
I20250114 02:55:14.127395 3855 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 02:55:14.128753 3751 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250114 02:55:14.129138 3751 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-2/data,/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-2/wal
uuid: "f9a6e2eab6e541eca12ee79c1d3004e8"
format_stamp: "Formatted at 2025-01-14 02:55:01 on dist-test-slave-36vg"
I20250114 02:55:14.131462 3751 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 02:55:14.182112 3751 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 02:55:14.183575 3751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 02:55:14.183993 3751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 02:55:14.256937 3751 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.124:44431
I20250114 02:55:14.257020 3906 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.124:44431 every 8 connection(s)
I20250114 02:55:14.259788 3751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-2/data/info.pb
I20250114 02:55:14.264400 429 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrCUXNq/build/tsan/bin/kudu as pid 3751
I20250114 02:55:14.272663 3907 sys_catalog.cc:263] Verifying existing consensus state
I20250114 02:55:14.281060 3907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8: Bootstrap starting.
I20250114 02:55:14.296797 3907 log.cc:826] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8: Log is configured to *not* fsync() on all Append() calls
I20250114 02:55:14.311726 3907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250114 02:55:14.312590 3907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8: Bootstrap complete.
I20250114 02:55:14.317574 3912 raft_consensus.cc:491] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a8f3d9567c014741a7e628bab3e18e27)
I20250114 02:55:14.317956 3912 raft_consensus.cc:513] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:55:14.319428 3912 leader_election.cc:290] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235), f9a6e2eab6e541eca12ee79c1d3004e8 (127.0.107.124:44431)
W20250114 02:55:14.326251 3045 leader_election.cc:336] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235): Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111)
I20250114 02:55:14.336966 3907 raft_consensus.cc:357] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:55:14.339571 3907 raft_consensus.cc:738] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9a6e2eab6e541eca12ee79c1d3004e8, State: Initialized, Role: FOLLOWER
I20250114 02:55:14.340478 3907 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:55:14.343379 3916 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } } }
I20250114 02:55:14.344401 3916 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [sys.catalog]: This master's current role is: FOLLOWER
I20250114 02:55:14.345314 3907 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [sys.catalog]: configured and running, proceeding with master startup.
I20250114 02:55:14.345456 3882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" candidate_term: 3 candidate_status { last_received { term: 2 index: 19 } } ignore_live_leader: false dest_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" is_pre_election: true
I20250114 02:55:14.346201 3882 raft_consensus.cc:2463] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfb2e2fecfb94af8bf892c6f774c2d89 in term 1.
I20250114 02:55:14.347860 3046 leader_election.cc:304] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bfb2e2fecfb94af8bf892c6f774c2d89, f9a6e2eab6e541eca12ee79c1d3004e8; no voters: a8f3d9567c014741a7e628bab3e18e27
I20250114 02:55:14.349013 3912 raft_consensus.cc:2798] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 02:55:14.349372 3912 raft_consensus.cc:491] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [term 2 FOLLOWER]: Starting leader election (detected failure of leader a8f3d9567c014741a7e628bab3e18e27)
I20250114 02:55:14.349690 3912 raft_consensus.cc:3054] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [term 2 FOLLOWER]: Advancing to term 3
I20250114 02:55:14.356518 3912 raft_consensus.cc:513] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:55:14.360002 3912 leader_election.cc:290] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [CANDIDATE]: Term 3 election: Requested vote from peers a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235), f9a6e2eab6e541eca12ee79c1d3004e8 (127.0.107.124:44431)
W20250114 02:55:14.362644 3045 leader_election.cc:336] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235): Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111)
I20250114 02:55:14.362553 3882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" candidate_term: 3 candidate_status { last_received { term: 2 index: 19 } } ignore_live_leader: false dest_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8"
I20250114 02:55:14.363198 3882 raft_consensus.cc:3054] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 1 FOLLOWER]: Advancing to term 3
I20250114 02:55:14.370301 3882 raft_consensus.cc:2463] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfb2e2fecfb94af8bf892c6f774c2d89 in term 3.
I20250114 02:55:14.371292 3046 leader_election.cc:304] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bfb2e2fecfb94af8bf892c6f774c2d89, f9a6e2eab6e541eca12ee79c1d3004e8; no voters: a8f3d9567c014741a7e628bab3e18e27
I20250114 02:55:14.371928 3912 raft_consensus.cc:2798] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 02:55:14.373178 3912 raft_consensus.cc:695] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [term 3 LEADER]: Becoming Leader. State: Replica: bfb2e2fecfb94af8bf892c6f774c2d89, State: Running, Role: LEADER
I20250114 02:55:14.373929 3912 consensus_queue.cc:237] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 2.19, Last appended by leader: 19, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:55:14.377882 3927 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bfb2e2fecfb94af8bf892c6f774c2d89. Latest consensus state: current_term: 3 leader_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } } }
I20250114 02:55:14.378695 3927 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [sys.catalog]: This master's current role is: LEADER
I20250114 02:55:14.381875 3932 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250114 02:55:14.400616 3932 catalog_manager.cc:670] Loaded metadata for table default.table_b4cf0d29abe14186b59a52f5bcf7d11d [id=0b4964a250114a8f855dddd9ba3b6f76]
I20250114 02:55:14.401223 3934 catalog_manager.cc:1260] Loaded cluster ID: 383d09ac0a7b47d1bf47222602c9cf34
I20250114 02:55:14.401629 3934 catalog_manager.cc:1553] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8: loading cluster ID for follower catalog manager: success
I20250114 02:55:14.403673 3932 catalog_manager.cc:670] Loaded metadata for table default.table_8aa1373bd9824b1a8cbed27ede045e55 [id=5dbcc4cbd424444490d042e890c01b93]
I20250114 02:55:14.405390 3932 catalog_manager.cc:670] Loaded metadata for table default.table_bab0c0670ef34820b254580eeb9eef70 [id=89d59ce4ef604cce9b1e9a01a4aea791]
I20250114 02:55:14.409586 3934 catalog_manager.cc:1575] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8: acquiring CA information for follower catalog manager: success
I20250114 02:55:14.415340 3932 tablet_loader.cc:96] loaded metadata for tablet 5e5a4f58a4ba47e59477208eb2bd833b (table default.table_bab0c0670ef34820b254580eeb9eef70 [id=89d59ce4ef604cce9b1e9a01a4aea791])
I20250114 02:55:14.416761 3932 tablet_loader.cc:96] loaded metadata for tablet 688cd8217fb04e4e930cb2421bfcfd9b (table default.table_8aa1373bd9824b1a8cbed27ede045e55 [id=5dbcc4cbd424444490d042e890c01b93])
I20250114 02:55:14.417934 3932 tablet_loader.cc:96] loaded metadata for tablet 83abece59e0541f1ad0eefee1a8e7994 (table default.table_b4cf0d29abe14186b59a52f5bcf7d11d [id=0b4964a250114a8f855dddd9ba3b6f76])
I20250114 02:55:14.427933 3536 heartbeater.cc:346] Connected to a master server at 127.0.107.124:44431
I20250114 02:55:14.429594 3934 catalog_manager.cc:1603] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250114 02:55:14.394516 3751 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 02:55:14.432164 3751 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250114 02:55:14.434345 3932 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250114 02:55:14.437148 3872 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "39c64942315c4716baaee0c251c05acd" instance_seqno: 1736823307893419) as {username='slave'} at 127.0.107.66:52389; Asking this server to re-register.
I20250114 02:55:14.438724 3536 heartbeater.cc:463] Registering TS with master...
I20250114 02:55:14.439868 3932 catalog_manager.cc:1260] Loaded cluster ID: 383d09ac0a7b47d1bf47222602c9cf34
I20250114 02:55:14.440177 3932 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250114 02:55:14.445456 3932 catalog_manager.cc:1505] Loading token signing keys...
I20250114 02:55:14.444001 3872 ts_manager.cc:194] Registered new tserver with Master: 39c64942315c4716baaee0c251c05acd (127.0.107.66:33487)
I20250114 02:55:14.450137 3932 catalog_manager.cc:5910] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89: Loaded TSK: 0
I20250114 02:55:14.451627 3932 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250114 02:55:14.456290 3932 catalog_manager.cc:1528] Loading latest processed Hive Metastore notification log event ID...
I20250114 02:55:14.460681 3932 catalog_manager.cc:5748] Last processed Hive Metastore notification event ID: -1
I20250114 02:55:14.473256 3678 heartbeater.cc:346] Connected to a master server at 127.0.107.124:44431
I20250114 02:55:14.473811 3678 heartbeater.cc:502] Master 127.0.107.124:44431 was elected leader, sending a full tablet report...
I20250114 02:55:14.475914 3392 heartbeater.cc:346] Connected to a master server at 127.0.107.124:44431
I20250114 02:55:14.478672 3872 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" instance_seqno: 1736823309820499) as {username='slave'} at 127.0.107.67:50003; Asking this server to re-register.
I20250114 02:55:14.481470 3678 heartbeater.cc:463] Registering TS with master...
I20250114 02:55:14.481392 3871 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" instance_seqno: 1736823305980627) as {username='slave'} at 127.0.107.65:46943; Asking this server to re-register.
I20250114 02:55:14.483563 3392 heartbeater.cc:463] Registering TS with master...
I20250114 02:55:14.485651 3871 ts_manager.cc:194] Registered new tserver with Master: 5017c179ddc640fc9ebf5dea7f3957f8 (127.0.107.67:45409)
I20250114 02:55:14.487439 3872 ts_manager.cc:194] Registered new tserver with Master: 7b7de0f9038c4ba287750ad132cb3597 (127.0.107.65:34329)
I20250114 02:55:14.505016 3059 master_service.cc:946] ReplaceTablet: received request to replace tablet 5e5a4f58a4ba47e59477208eb2bd833b from {username='slave'} at 127.0.0.1:42018
I20250114 02:55:14.507515 3054 master_service.cc:946] ReplaceTablet: received request to replace tablet 688cd8217fb04e4e930cb2421bfcfd9b from {username='slave'} at 127.0.0.1:42018
I20250114 02:55:14.510385 3057 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42018:
name: "default.Table_c27b5989e4ba41f285a14120cb0dba75"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:55:14.511953 3882 raft_consensus.cc:1270] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Refusing update from remote peer bfb2e2fecfb94af8bf892c6f774c2d89: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 21. (index mismatch)
I20250114 02:55:14.512157 3056 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42018:
table { table_name: "default.Table_8aa1373bd9824b1a8cbed27ede045e55" } modify_external_catalogs: true
I20250114 02:55:14.512967 3056 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42018:
table { table_name: "default.Table_8aa1373bd9824b1a8cbed27ede045e55" } modify_external_catalogs: true
I20250114 02:55:14.513754 3053 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42018:
name: "default.Table_1e2047182d2241f4b619010ca10e11fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:55:14.514874 3050 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42018:
table { table_name: "default.Table_bab0c0670ef34820b254580eeb9eef70" } new_table_name: "default.Table_91d61e17dfec48479865a7feb0460bd9" modify_external_catalogs: true
I20250114 02:55:14.516489 3051 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42018:
name: "default.Table_f39207a74e3a4f68acdb67e4a70f7510"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 02:55:14.518091 3052 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42018:
name: "default.Table_6e897f928c824eedbe6f75b0e4ad6b93"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20250114 02:55:14.519372 3247 hms_notification_log_listener.cc:263] Received out-of-order notification log event (last processed event ID: 1): 1 CREATE_TABLE default.table_627b0faab8eb44c58814768e6dd28c15
*** Check failure stack trace: ***
I20250114 02:55:14.523090 3927 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 4, Time since last communication: 0.000s
I20250114 02:55:14.524013 3055 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42018:
table { table_name: "default.Table_b4cf0d29abe14186b59a52f5bcf7d11d" } new_table_name: "default.Table_abbbf6099c5b4880ad06b2e250de4248" modify_external_catalogs: true
W20250114 02:55:14.525135 3045 consensus_peers.cc:487] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 -> Peer a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235): Couldn't send request to peer a8f3d9567c014741a7e628bab3e18e27. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
@ 0x7f679df89950 google::LogMessage::Flush() at ??:0
@ 0x7f679df896bf google::LogMessage::~LogMessage() at ??:0
I20250114 02:55:14.565600 3916 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bfb2e2fecfb94af8bf892c6f774c2d89. Latest consensus state: current_term: 3 leader_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } } }
I20250114 02:55:14.569532 3943 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } } }
I20250114 02:55:14.570612 3943 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [sys.catalog]: This master's current role is: FOLLOWER
I20250114 02:55:14.566442 3916 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [sys.catalog]: This master's current role is: FOLLOWER
I20250114 02:55:14.575985 3939 mvcc.cc:204] Tried to move back new op lower bound from 7114028296211054592 to 7114028295682777088. Current Snapshot: MvccSnapshot[applied={T|T < 7114028296211054592}]
I20250114 02:55:14.581048 3916 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } } }
I20250114 02:55:14.581806 3916 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [sys.catalog]: This master's current role is: FOLLOWER
I20250114 02:55:14.586735 3912 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } } }
I20250114 02:55:14.587492 3912 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89 [sys.catalog]: This master's current role is: LEADER
I20250114 02:55:14.589808 3058 catalog_manager.cc:5526] T 5e5a4f58a4ba47e59477208eb2bd833b P 7b7de0f9038c4ba287750ad132cb3597 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b7de0f9038c4ba287750ad132cb3597 (127.0.107.65). New cstate: current_term: 1 leader_uuid: "7b7de0f9038c4ba287750ad132cb3597" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } health_report { overall_health: HEALTHY } } }
@ 0x7f67a8e8382a kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f67a8e8307c kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250114 02:55:14.673162 3327 tablet_service.cc:1467] Processing CreateTablet for tablet 83abece59e0541f1ad0eefee1a8e7994 (DEFAULT_TABLE table=default.table_b4cf0d29abe14186b59a52f5bcf7d11d [id=0b4964a250114a8f855dddd9ba3b6f76]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:55:14.674546 3327 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83abece59e0541f1ad0eefee1a8e7994. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:55:14.675881 3054 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89: Sending DeleteTablet for 3 replicas of tablet 688cd8217fb04e4e930cb2421bfcfd9b
I20250114 02:55:14.679298 3471 tablet_service.cc:1467] Processing CreateTablet for tablet 83abece59e0541f1ad0eefee1a8e7994 (DEFAULT_TABLE table=default.table_b4cf0d29abe14186b59a52f5bcf7d11d [id=0b4964a250114a8f855dddd9ba3b6f76]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:55:14.680536 3471 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83abece59e0541f1ad0eefee1a8e7994. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:55:14.680914 3054 catalog_manager.cc:6596] ReplaceTablet: tablet 688cd8217fb04e4e930cb2421bfcfd9b deleted and replaced by tablet 7441da6add1a46e7b18148ea27d05323
I20250114 02:55:14.683113 3470 tablet_service.cc:1514] Processing DeleteTablet for tablet 688cd8217fb04e4e930cb2421bfcfd9b with delete_type TABLET_DATA_DELETED (replaced by tablet 7441da6add1a46e7b18148ea27d05323) from {username='slave'} at 127.0.0.1:58624
I20250114 02:55:14.687247 3326 tablet_service.cc:1514] Processing DeleteTablet for tablet 688cd8217fb04e4e930cb2421bfcfd9b with delete_type TABLET_DATA_DELETED (replaced by tablet 7441da6add1a46e7b18148ea27d05323) from {username='slave'} at 127.0.0.1:58840
I20250114 02:55:14.700891 3955 tablet_replica.cc:331] T 688cd8217fb04e4e930cb2421bfcfd9b P 39c64942315c4716baaee0c251c05acd: stopping tablet replica
I20250114 02:55:14.701673 3955 raft_consensus.cc:2238] T 688cd8217fb04e4e930cb2421bfcfd9b P 39c64942315c4716baaee0c251c05acd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:55:14.702198 3955 raft_consensus.cc:2267] T 688cd8217fb04e4e930cb2421bfcfd9b P 39c64942315c4716baaee0c251c05acd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:55:14.704063 3955 ts_tablet_manager.cc:1905] T 688cd8217fb04e4e930cb2421bfcfd9b P 39c64942315c4716baaee0c251c05acd: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f67a8e86d32 kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20250114 02:55:14.718890 3613 tablet_service.cc:1467] Processing CreateTablet for tablet 83abece59e0541f1ad0eefee1a8e7994 (DEFAULT_TABLE table=default.table_b4cf0d29abe14186b59a52f5bcf7d11d [id=0b4964a250114a8f855dddd9ba3b6f76]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:55:14.719983 3613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83abece59e0541f1ad0eefee1a8e7994. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:55:14.720479 3612 tablet_service.cc:1514] Processing DeleteTablet for tablet 688cd8217fb04e4e930cb2421bfcfd9b with delete_type TABLET_DATA_DELETED (replaced by tablet 7441da6add1a46e7b18148ea27d05323) from {username='slave'} at 127.0.0.1:60676
I20250114 02:55:14.729326 3940 mvcc.cc:204] Tried to move back new op lower bound from 7114028296211054592 to 7114028279235149824. Current Snapshot: MvccSnapshot[applied={T|T < 7114028296211054592 or (T in {7114028296211054592})}]
I20250114 02:55:14.738731 3059 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89: Sending DeleteTablet for 3 replicas of tablet 5e5a4f58a4ba47e59477208eb2bd833b
I20250114 02:55:14.740376 3059 catalog_manager.cc:6596] ReplaceTablet: tablet 5e5a4f58a4ba47e59477208eb2bd833b deleted and replaced by tablet a02695476ce44eea9bc7a53871bc45a0
I20250114 02:55:14.742779 3960 tablet_bootstrap.cc:492] T 83abece59e0541f1ad0eefee1a8e7994 P 5017c179ddc640fc9ebf5dea7f3957f8: Bootstrap starting.
I20250114 02:55:14.744020 3956 tablet_replica.cc:331] T 688cd8217fb04e4e930cb2421bfcfd9b P 7b7de0f9038c4ba287750ad132cb3597: stopping tablet replica
I20250114 02:55:14.746954 3613 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e5a4f58a4ba47e59477208eb2bd833b with delete_type TABLET_DATA_DELETED (replaced by tablet a02695476ce44eea9bc7a53871bc45a0) from {username='slave'} at 127.0.0.1:60676
I20250114 02:55:14.747140 3956 raft_consensus.cc:2238] T 688cd8217fb04e4e930cb2421bfcfd9b P 7b7de0f9038c4ba287750ad132cb3597 [term 1 LEADER]: Raft consensus shutting down.
I20250114 02:55:14.748245 3956 raft_consensus.cc:2267] T 688cd8217fb04e4e930cb2421bfcfd9b P 7b7de0f9038c4ba287750ad132cb3597 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:55:14.749787 3325 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e5a4f58a4ba47e59477208eb2bd833b with delete_type TABLET_DATA_DELETED (replaced by tablet a02695476ce44eea9bc7a53871bc45a0) from {username='slave'} at 127.0.0.1:58840
I20250114 02:55:14.751683 3470 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e5a4f58a4ba47e59477208eb2bd833b with delete_type TABLET_DATA_DELETED (replaced by tablet a02695476ce44eea9bc7a53871bc45a0) from {username='slave'} at 127.0.0.1:58624
I20250114 02:55:14.752717 3959 tablet_replica.cc:331] T 688cd8217fb04e4e930cb2421bfcfd9b P 5017c179ddc640fc9ebf5dea7f3957f8: stopping tablet replica
I20250114 02:55:14.758003 3959 raft_consensus.cc:2238] T 688cd8217fb04e4e930cb2421bfcfd9b P 5017c179ddc640fc9ebf5dea7f3957f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:55:14.761171 3960 tablet_bootstrap.cc:654] T 83abece59e0541f1ad0eefee1a8e7994 P 5017c179ddc640fc9ebf5dea7f3957f8: Neither blocks nor log segments found. Creating new log.
I20250114 02:55:14.761680 3959 raft_consensus.cc:2267] T 688cd8217fb04e4e930cb2421bfcfd9b P 5017c179ddc640fc9ebf5dea7f3957f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:55:14.762626 3956 ts_tablet_manager.cc:1905] T 688cd8217fb04e4e930cb2421bfcfd9b P 7b7de0f9038c4ba287750ad132cb3597: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:55:14.764652 3959 ts_tablet_manager.cc:1905] T 688cd8217fb04e4e930cb2421bfcfd9b P 5017c179ddc640fc9ebf5dea7f3957f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:55:14.771028 3957 tablet_bootstrap.cc:492] T 83abece59e0541f1ad0eefee1a8e7994 P 39c64942315c4716baaee0c251c05acd: Bootstrap starting.
I20250114 02:55:14.777067 3961 tablet_bootstrap.cc:492] T 83abece59e0541f1ad0eefee1a8e7994 P 7b7de0f9038c4ba287750ad132cb3597: Bootstrap starting.
@ 0x7f67a8e86cea _ZNSt3__18__invokeIRZN4kudu6master30HmsNotificationLogListenerTask4InitEvE3$_0JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
I20250114 02:55:14.782284 3957 tablet_bootstrap.cc:654] T 83abece59e0541f1ad0eefee1a8e7994 P 39c64942315c4716baaee0c251c05acd: Neither blocks nor log segments found. Creating new log.
I20250114 02:55:14.785745 3961 tablet_bootstrap.cc:654] T 83abece59e0541f1ad0eefee1a8e7994 P 7b7de0f9038c4ba287750ad132cb3597: Neither blocks nor log segments found. Creating new log.
I20250114 02:55:14.792701 3327 tablet_service.cc:1467] Processing CreateTablet for tablet 7441da6add1a46e7b18148ea27d05323 (DEFAULT_TABLE table=default.table_8aa1373bd9824b1a8cbed27ede045e55 [id=5dbcc4cbd424444490d042e890c01b93]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:55:14.793776 3327 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7441da6add1a46e7b18148ea27d05323. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:55:14.792501 3471 tablet_service.cc:1467] Processing CreateTablet for tablet 7441da6add1a46e7b18148ea27d05323 (DEFAULT_TABLE table=default.table_8aa1373bd9824b1a8cbed27ede045e55 [id=5dbcc4cbd424444490d042e890c01b93]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:55:14.807965 3471 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7441da6add1a46e7b18148ea27d05323. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:55:14.797111 3612 tablet_service.cc:1467] Processing CreateTablet for tablet 7441da6add1a46e7b18148ea27d05323 (DEFAULT_TABLE table=default.table_8aa1373bd9824b1a8cbed27ede045e55 [id=5dbcc4cbd424444490d042e890c01b93]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:55:14.808779 3960 tablet_bootstrap.cc:492] T 83abece59e0541f1ad0eefee1a8e7994 P 5017c179ddc640fc9ebf5dea7f3957f8: No bootstrap required, opened a new log
I20250114 02:55:14.810096 3960 ts_tablet_manager.cc:1397] T 83abece59e0541f1ad0eefee1a8e7994 P 5017c179ddc640fc9ebf5dea7f3957f8: Time spent bootstrapping tablet: real 0.068s user 0.016s sys 0.002s
I20250114 02:55:14.810398 3612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7441da6add1a46e7b18148ea27d05323. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:55:14.812589 3960 raft_consensus.cc:357] T 83abece59e0541f1ad0eefee1a8e7994 P 5017c179ddc640fc9ebf5dea7f3957f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:14.814343 3960 raft_consensus.cc:383] T 83abece59e0541f1ad0eefee1a8e7994 P 5017c179ddc640fc9ebf5dea7f3957f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:55:14.814659 3960 raft_consensus.cc:738] T 83abece59e0541f1ad0eefee1a8e7994 P 5017c179ddc640fc9ebf5dea7f3957f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5017c179ddc640fc9ebf5dea7f3957f8, State: Initialized, Role: FOLLOWER
I20250114 02:55:14.815425 3960 consensus_queue.cc:260] T 83abece59e0541f1ad0eefee1a8e7994 P 5017c179ddc640fc9ebf5dea7f3957f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:14.790889 3955 ts_tablet_manager.cc:1918] T 688cd8217fb04e4e930cb2421bfcfd9b P 39c64942315c4716baaee0c251c05acd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:55:14.820662 3955 log.cc:1198] T 688cd8217fb04e4e930cb2421bfcfd9b P 39c64942315c4716baaee0c251c05acd: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/688cd8217fb04e4e930cb2421bfcfd9b
I20250114 02:55:14.822695 3961 tablet_bootstrap.cc:492] T 83abece59e0541f1ad0eefee1a8e7994 P 7b7de0f9038c4ba287750ad132cb3597: No bootstrap required, opened a new log
I20250114 02:55:14.823446 3961 ts_tablet_manager.cc:1397] T 83abece59e0541f1ad0eefee1a8e7994 P 7b7de0f9038c4ba287750ad132cb3597: Time spent bootstrapping tablet: real 0.047s user 0.010s sys 0.004s
I20250114 02:55:14.824903 3955 ts_tablet_manager.cc:1939] T 688cd8217fb04e4e930cb2421bfcfd9b P 39c64942315c4716baaee0c251c05acd: Deleting consensus metadata
I20250114 02:55:14.827819 3044 catalog_manager.cc:4872] TS 39c64942315c4716baaee0c251c05acd (127.0.107.66:33487): tablet 688cd8217fb04e4e930cb2421bfcfd9b (table default.table_8aa1373bd9824b1a8cbed27ede045e55 [id=5dbcc4cbd424444490d042e890c01b93]) successfully deleted
I20250114 02:55:14.829722 3679 heartbeater.cc:502] Master 127.0.107.125:38653 was elected leader, sending a full tablet report...
I20250114 02:55:14.826407 3961 raft_consensus.cc:357] T 83abece59e0541f1ad0eefee1a8e7994 P 7b7de0f9038c4ba287750ad132cb3597 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:14.830835 3961 raft_consensus.cc:383] T 83abece59e0541f1ad0eefee1a8e7994 P 7b7de0f9038c4ba287750ad132cb3597 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:55:14.831104 3961 raft_consensus.cc:738] T 83abece59e0541f1ad0eefee1a8e7994 P 7b7de0f9038c4ba287750ad132cb3597 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b7de0f9038c4ba287750ad132cb3597, State: Initialized, Role: FOLLOWER
I20250114 02:55:14.832930 3960 ts_tablet_manager.cc:1428] T 83abece59e0541f1ad0eefee1a8e7994 P 5017c179ddc640fc9ebf5dea7f3957f8: Time spent starting tablet: real 0.023s user 0.002s sys 0.005s
@ 0x7f67a8e86c7a std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
I20250114 02:55:14.833736 3961 consensus_queue.cc:260] T 83abece59e0541f1ad0eefee1a8e7994 P 7b7de0f9038c4ba287750ad132cb3597 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:14.839144 3393 heartbeater.cc:502] Master 127.0.107.125:38653 was elected leader, sending a full tablet report...
I20250114 02:55:14.847932 3955 tablet_replica.cc:331] T 5e5a4f58a4ba47e59477208eb2bd833b P 39c64942315c4716baaee0c251c05acd: stopping tablet replica
I20250114 02:55:14.850389 3961 ts_tablet_manager.cc:1428] T 83abece59e0541f1ad0eefee1a8e7994 P 7b7de0f9038c4ba287750ad132cb3597: Time spent starting tablet: real 0.027s user 0.001s sys 0.004s
I20250114 02:55:14.857622 3613 tablet_service.cc:1467] Processing CreateTablet for tablet a02695476ce44eea9bc7a53871bc45a0 (DEFAULT_TABLE table=default.table_bab0c0670ef34820b254580eeb9eef70 [id=89d59ce4ef604cce9b1e9a01a4aea791]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:55:14.858736 3613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a02695476ce44eea9bc7a53871bc45a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:55:14.861002 3326 tablet_service.cc:1467] Processing CreateTablet for tablet a02695476ce44eea9bc7a53871bc45a0 (DEFAULT_TABLE table=default.table_bab0c0670ef34820b254580eeb9eef70 [id=89d59ce4ef604cce9b1e9a01a4aea791]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:55:14.862140 3326 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a02695476ce44eea9bc7a53871bc45a0. 1 dirs total, 0 dirs full, 0 dirs failed
W20250114 02:55:14.867344 3059 catalog_manager.cc:5359] Ignoring report from unknown tablet c174bcbc5d51402b8ae332b87ffa2139
I20250114 02:55:14.869968 3470 tablet_service.cc:1467] Processing CreateTablet for tablet a02695476ce44eea9bc7a53871bc45a0 (DEFAULT_TABLE table=default.table_bab0c0670ef34820b254580eeb9eef70 [id=89d59ce4ef604cce9b1e9a01a4aea791]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 02:55:14.871620 3960 tablet_bootstrap.cc:492] T 7441da6add1a46e7b18148ea27d05323 P 5017c179ddc640fc9ebf5dea7f3957f8: Bootstrap starting.
I20250114 02:55:14.871929 3470 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a02695476ce44eea9bc7a53871bc45a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 02:55:14.879627 3960 tablet_bootstrap.cc:654] T 7441da6add1a46e7b18148ea27d05323 P 5017c179ddc640fc9ebf5dea7f3957f8: Neither blocks nor log segments found. Creating new log.
I20250114 02:55:14.890053 3058 master_service.cc:946] ReplaceTablet: received request to replace tablet 5e5a4f58a4ba47e59477208eb2bd833b from {username='slave'} at 127.0.0.1:42018
I20250114 02:55:14.893281 3959 ts_tablet_manager.cc:1918] T 688cd8217fb04e4e930cb2421bfcfd9b P 5017c179ddc640fc9ebf5dea7f3957f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:55:14.893949 3956 ts_tablet_manager.cc:1918] T 688cd8217fb04e4e930cb2421bfcfd9b P 7b7de0f9038c4ba287750ad132cb3597: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:55:14.895272 3956 log.cc:1198] T 688cd8217fb04e4e930cb2421bfcfd9b P 7b7de0f9038c4ba287750ad132cb3597: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/688cd8217fb04e4e930cb2421bfcfd9b
I20250114 02:55:14.897009 3956 ts_tablet_manager.cc:1939] T 688cd8217fb04e4e930cb2421bfcfd9b P 7b7de0f9038c4ba287750ad132cb3597: Deleting consensus metadata
I20250114 02:55:14.900705 3043 catalog_manager.cc:4872] TS 7b7de0f9038c4ba287750ad132cb3597 (127.0.107.65:34329): tablet 688cd8217fb04e4e930cb2421bfcfd9b (table default.table_8aa1373bd9824b1a8cbed27ede045e55 [id=5dbcc4cbd424444490d042e890c01b93]) successfully deleted
W20250114 02:55:14.902521 3054 catalog_manager.cc:5359] Ignoring report from unknown tablet c174bcbc5d51402b8ae332b87ffa2139
I20250114 02:55:14.902498 3959 log.cc:1198] T 688cd8217fb04e4e930cb2421bfcfd9b P 5017c179ddc640fc9ebf5dea7f3957f8: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/688cd8217fb04e4e930cb2421bfcfd9b
I20250114 02:55:14.903380 3959 ts_tablet_manager.cc:1939] T 688cd8217fb04e4e930cb2421bfcfd9b P 5017c179ddc640fc9ebf5dea7f3957f8: Deleting consensus metadata
I20250114 02:55:14.911962 3043 catalog_manager.cc:4872] TS 5017c179ddc640fc9ebf5dea7f3957f8 (127.0.107.67:45409): tablet 688cd8217fb04e4e930cb2421bfcfd9b (table default.table_8aa1373bd9824b1a8cbed27ede045e55 [id=5dbcc4cbd424444490d042e890c01b93]) successfully deleted
I20250114 02:55:14.914958 3956 tablet_replica.cc:331] T 5e5a4f58a4ba47e59477208eb2bd833b P 7b7de0f9038c4ba287750ad132cb3597: stopping tablet replica
I20250114 02:55:14.915757 3956 raft_consensus.cc:2238] T 5e5a4f58a4ba47e59477208eb2bd833b P 7b7de0f9038c4ba287750ad132cb3597 [term 1 LEADER]: Raft consensus shutting down.
I20250114 02:55:14.917372 3959 tablet_replica.cc:331] T 5e5a4f58a4ba47e59477208eb2bd833b P 5017c179ddc640fc9ebf5dea7f3957f8: stopping tablet replica
@ 0x7f67a8e86c42 std::__1::__function::__alloc_func<>::operator()() at ??:0
I20250114 02:55:14.918252 3956 raft_consensus.cc:2267] T 5e5a4f58a4ba47e59477208eb2bd833b P 7b7de0f9038c4ba287750ad132cb3597 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:55:14.920101 3956 ts_tablet_manager.cc:1905] T 5e5a4f58a4ba47e59477208eb2bd833b P 7b7de0f9038c4ba287750ad132cb3597: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:55:14.922679 3612 tablet_service.cc:1514] Processing DeleteTablet for tablet 688cd8217fb04e4e930cb2421bfcfd9b with delete_type TABLET_DATA_DELETED (replaced by tablet 7441da6add1a46e7b18148ea27d05323) from {username='slave'} at 127.0.0.1:60676
I20250114 02:55:14.924655 3959 raft_consensus.cc:2238] T 5e5a4f58a4ba47e59477208eb2bd833b P 5017c179ddc640fc9ebf5dea7f3957f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:55:14.925515 3959 raft_consensus.cc:2267] T 5e5a4f58a4ba47e59477208eb2bd833b P 5017c179ddc640fc9ebf5dea7f3957f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:55:14.928072 3959 ts_tablet_manager.cc:1905] T 5e5a4f58a4ba47e59477208eb2bd833b P 5017c179ddc640fc9ebf5dea7f3957f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:55:14.935462 3960 tablet_bootstrap.cc:492] T 7441da6add1a46e7b18148ea27d05323 P 5017c179ddc640fc9ebf5dea7f3957f8: No bootstrap required, opened a new log
I20250114 02:55:14.935873 3960 ts_tablet_manager.cc:1397] T 7441da6add1a46e7b18148ea27d05323 P 5017c179ddc640fc9ebf5dea7f3957f8: Time spent bootstrapping tablet: real 0.064s user 0.012s sys 0.035s
I20250114 02:55:14.939698 3325 tablet_service.cc:1514] Processing DeleteTablet for tablet 688cd8217fb04e4e930cb2421bfcfd9b with delete_type TABLET_DATA_DELETED (replaced by tablet 7441da6add1a46e7b18148ea27d05323) from {username='slave'} at 127.0.0.1:58840
I20250114 02:55:14.947364 3960 raft_consensus.cc:357] T 7441da6add1a46e7b18148ea27d05323 P 5017c179ddc640fc9ebf5dea7f3957f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:14.948097 3960 raft_consensus.cc:383] T 7441da6add1a46e7b18148ea27d05323 P 5017c179ddc640fc9ebf5dea7f3957f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:55:14.948415 3960 raft_consensus.cc:738] T 7441da6add1a46e7b18148ea27d05323 P 5017c179ddc640fc9ebf5dea7f3957f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5017c179ddc640fc9ebf5dea7f3957f8, State: Initialized, Role: FOLLOWER
I20250114 02:55:14.949059 3960 consensus_queue.cc:260] T 7441da6add1a46e7b18148ea27d05323 P 5017c179ddc640fc9ebf5dea7f3957f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:14.951352 3960 ts_tablet_manager.cc:1428] T 7441da6add1a46e7b18148ea27d05323 P 5017c179ddc640fc9ebf5dea7f3957f8: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
W20250114 02:55:14.952639 3058 catalog_manager.cc:5359] Ignoring report from unknown tablet c174bcbc5d51402b8ae332b87ffa2139
@ 0x7f67a8e85f3e std::__1::__function::__func<>::operator()() at ??:0
@ 0x59ea05 std::__1::__function::__value_func<>::operator()() at ??:?
@ 0x59e83a std::__1::function<>::operator()() at ??:?
I20250114 02:55:14.968462 3059 master_service.cc:946] ReplaceTablet: received request to replace tablet 83abece59e0541f1ad0eefee1a8e7994 from {username='slave'} at 127.0.0.1:42018
I20250114 02:55:14.983309 3959 ts_tablet_manager.cc:1918] T 5e5a4f58a4ba47e59477208eb2bd833b P 5017c179ddc640fc9ebf5dea7f3957f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:55:14.983731 3959 log.cc:1198] T 5e5a4f58a4ba47e59477208eb2bd833b P 5017c179ddc640fc9ebf5dea7f3957f8: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/5e5a4f58a4ba47e59477208eb2bd833b
I20250114 02:55:14.984470 3959 ts_tablet_manager.cc:1939] T 5e5a4f58a4ba47e59477208eb2bd833b P 5017c179ddc640fc9ebf5dea7f3957f8: Deleting consensus metadata
I20250114 02:55:14.986979 3043 catalog_manager.cc:4872] TS 5017c179ddc640fc9ebf5dea7f3957f8 (127.0.107.67:45409): tablet 5e5a4f58a4ba47e59477208eb2bd833b (table default.table_bab0c0670ef34820b254580eeb9eef70 [id=89d59ce4ef604cce9b1e9a01a4aea791]) successfully deleted
@ 0x7f67a01ccfe7 kudu::Thread::SuperviseThread() at ??:0
@ 0x468a00 __tsan_thread_start_func at ??:?
I20250114 02:55:14.991314 3956 ts_tablet_manager.cc:1918] T 5e5a4f58a4ba47e59477208eb2bd833b P 7b7de0f9038c4ba287750ad132cb3597: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:55:14.991796 3956 log.cc:1198] T 5e5a4f58a4ba47e59477208eb2bd833b P 7b7de0f9038c4ba287750ad132cb3597: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/5e5a4f58a4ba47e59477208eb2bd833b
I20250114 02:55:14.992519 3956 ts_tablet_manager.cc:1939] T 5e5a4f58a4ba47e59477208eb2bd833b P 7b7de0f9038c4ba287750ad132cb3597: Deleting consensus metadata
W20250114 02:55:14.996536 3043 catalog_manager.cc:4839] TS 5017c179ddc640fc9ebf5dea7f3957f8 (127.0.107.67:45409): delete failed for tablet 688cd8217fb04e4e930cb2421bfcfd9b because the tablet was not found. No further retry: Not found: Tablet not found: 688cd8217fb04e4e930cb2421bfcfd9b
I20250114 02:55:15.001605 3957 tablet_bootstrap.cc:492] T 83abece59e0541f1ad0eefee1a8e7994 P 39c64942315c4716baaee0c251c05acd: No bootstrap required, opened a new log
I20250114 02:55:15.002236 3957 ts_tablet_manager.cc:1397] T 83abece59e0541f1ad0eefee1a8e7994 P 39c64942315c4716baaee0c251c05acd: Time spent bootstrapping tablet: real 0.232s user 0.008s sys 0.006s
I20250114 02:55:14.997251 3043 catalog_manager.cc:4872] TS 7b7de0f9038c4ba287750ad132cb3597 (127.0.107.65:34329): tablet 5e5a4f58a4ba47e59477208eb2bd833b (table default.table_bab0c0670ef34820b254580eeb9eef70 [id=89d59ce4ef604cce9b1e9a01a4aea791]) successfully deleted
W20250114 02:55:15.004906 3043 catalog_manager.cc:4839] TS 7b7de0f9038c4ba287750ad132cb3597 (127.0.107.65:34329): delete failed for tablet 688cd8217fb04e4e930cb2421bfcfd9b because the tablet was not found. No further retry: Not found: Tablet not found: 688cd8217fb04e4e930cb2421bfcfd9b
I20250114 02:55:15.006997 3957 raft_consensus.cc:357] T 83abece59e0541f1ad0eefee1a8e7994 P 39c64942315c4716baaee0c251c05acd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:15.007683 3957 raft_consensus.cc:383] T 83abece59e0541f1ad0eefee1a8e7994 P 39c64942315c4716baaee0c251c05acd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:55:15.007936 3957 raft_consensus.cc:738] T 83abece59e0541f1ad0eefee1a8e7994 P 39c64942315c4716baaee0c251c05acd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39c64942315c4716baaee0c251c05acd, State: Initialized, Role: FOLLOWER
I20250114 02:55:15.008572 3957 consensus_queue.cc:260] T 83abece59e0541f1ad0eefee1a8e7994 P 39c64942315c4716baaee0c251c05acd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
@ 0x7f67a184c6db start_thread at ??:0
I20250114 02:55:15.024248 3961 tablet_bootstrap.cc:492] T a02695476ce44eea9bc7a53871bc45a0 P 7b7de0f9038c4ba287750ad132cb3597: Bootstrap starting.
I20250114 02:55:15.032286 3961 tablet_bootstrap.cc:654] T a02695476ce44eea9bc7a53871bc45a0 P 7b7de0f9038c4ba287750ad132cb3597: Neither blocks nor log segments found. Creating new log.
I20250114 02:55:15.036331 3059 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bfb2e2fecfb94af8bf892c6f774c2d89: Sending DeleteTablet for 3 replicas of tablet 83abece59e0541f1ad0eefee1a8e7994
I20250114 02:55:15.038826 3469 tablet_service.cc:1514] Processing DeleteTablet for tablet 83abece59e0541f1ad0eefee1a8e7994 with delete_type TABLET_DATA_DELETED (replaced by tablet 05f6bf25a81d46d3a79d334e3514ecfb) from {username='slave'} at 127.0.0.1:58624
I20250114 02:55:15.042258 3325 tablet_service.cc:1514] Processing DeleteTablet for tablet 83abece59e0541f1ad0eefee1a8e7994 with delete_type TABLET_DATA_DELETED (replaced by tablet 05f6bf25a81d46d3a79d334e3514ecfb) from {username='slave'} at 127.0.0.1:58840
I20250114 02:55:15.043036 3956 tablet_replica.cc:331] T 83abece59e0541f1ad0eefee1a8e7994 P 7b7de0f9038c4ba287750ad132cb3597: stopping tablet replica
I20250114 02:55:15.043774 3956 raft_consensus.cc:2238] T 83abece59e0541f1ad0eefee1a8e7994 P 7b7de0f9038c4ba287750ad132cb3597 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250114 02:55:15.044207 3956 raft_consensus.cc:2267] T 83abece59e0541f1ad0eefee1a8e7994 P 7b7de0f9038c4ba287750ad132cb3597 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250114 02:55:15.046463 3956 ts_tablet_manager.cc:1905] T 83abece59e0541f1ad0eefee1a8e7994 P 7b7de0f9038c4ba287750ad132cb3597: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:55:15.052377 3955 raft_consensus.cc:2238] T 5e5a4f58a4ba47e59477208eb2bd833b P 39c64942315c4716baaee0c251c05acd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250114 02:55:15.053072 3955 raft_consensus.cc:2267] T 5e5a4f58a4ba47e59477208eb2bd833b P 39c64942315c4716baaee0c251c05acd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 02:55:15.055464 3955 ts_tablet_manager.cc:1905] T 5e5a4f58a4ba47e59477208eb2bd833b P 39c64942315c4716baaee0c251c05acd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:55:15.060364 3325 tablet_service.cc:1514] Processing DeleteTablet for tablet 83abece59e0541f1ad0eefee1a8e7994 with delete_type TABLET_DATA_DELETED (replaced by tablet 05f6bf25a81d46d3a79d334e3514ecfb) from {username='slave'} at 127.0.0.1:58840
I20250114 02:55:15.064374 3612 tablet_service.cc:1514] Processing DeleteTablet for tablet 83abece59e0541f1ad0eefee1a8e7994 with delete_type TABLET_DATA_DELETED (replaced by tablet 05f6bf25a81d46d3a79d334e3514ecfb) from {username='slave'} at 127.0.0.1:60676
I20250114 02:55:15.065167 3959 tablet_replica.cc:331] T 83abece59e0541f1ad0eefee1a8e7994 P 5017c179ddc640fc9ebf5dea7f3957f8: stopping tablet replica
I20250114 02:55:15.065857 3059 catalog_manager.cc:6596] ReplaceTablet: tablet 83abece59e0541f1ad0eefee1a8e7994 deleted and replaced by tablet 05f6bf25a81d46d3a79d334e3514ecfb
I20250114 02:55:15.065873 3959 raft_consensus.cc:2238] T 83abece59e0541f1ad0eefee1a8e7994 P 5017c179ddc640fc9ebf5dea7f3957f8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250114 02:55:15.066320 3959 raft_consensus.cc:2267] T 83abece59e0541f1ad0eefee1a8e7994 P 5017c179ddc640fc9ebf5dea7f3957f8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250114 02:55:15.069960 3959 ts_tablet_manager.cc:1905] T 83abece59e0541f1ad0eefee1a8e7994 P 5017c179ddc640fc9ebf5dea7f3957f8: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f679cc2e71f clone at ??:0
I20250114 02:55:15.076849 3956 ts_tablet_manager.cc:1918] T 83abece59e0541f1ad0eefee1a8e7994 P 7b7de0f9038c4ba287750ad132cb3597: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:55:15.077243 3956 log.cc:1198] T 83abece59e0541f1ad0eefee1a8e7994 P 7b7de0f9038c4ba287750ad132cb3597: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-0/wal/wals/83abece59e0541f1ad0eefee1a8e7994
I20250114 02:55:15.077903 3956 ts_tablet_manager.cc:1939] T 83abece59e0541f1ad0eefee1a8e7994 P 7b7de0f9038c4ba287750ad132cb3597: Deleting consensus metadata
I20250114 02:55:15.079928 3960 tablet_bootstrap.cc:492] T a02695476ce44eea9bc7a53871bc45a0 P 5017c179ddc640fc9ebf5dea7f3957f8: Bootstrap starting.
I20250114 02:55:15.090039 3960 tablet_bootstrap.cc:654] T a02695476ce44eea9bc7a53871bc45a0 P 5017c179ddc640fc9ebf5dea7f3957f8: Neither blocks nor log segments found. Creating new log.
I20250114 02:55:15.097816 3959 ts_tablet_manager.cc:1918] T 83abece59e0541f1ad0eefee1a8e7994 P 5017c179ddc640fc9ebf5dea7f3957f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:55:15.097993 3961 tablet_bootstrap.cc:492] T a02695476ce44eea9bc7a53871bc45a0 P 7b7de0f9038c4ba287750ad132cb3597: No bootstrap required, opened a new log
I20250114 02:55:15.098176 3959 log.cc:1198] T 83abece59e0541f1ad0eefee1a8e7994 P 5017c179ddc640fc9ebf5dea7f3957f8: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-2/wal/wals/83abece59e0541f1ad0eefee1a8e7994
I20250114 02:55:15.098552 3961 ts_tablet_manager.cc:1397] T a02695476ce44eea9bc7a53871bc45a0 P 7b7de0f9038c4ba287750ad132cb3597: Time spent bootstrapping tablet: real 0.075s user 0.008s sys 0.004s
I20250114 02:55:15.098799 3959 ts_tablet_manager.cc:1939] T 83abece59e0541f1ad0eefee1a8e7994 P 5017c179ddc640fc9ebf5dea7f3957f8: Deleting consensus metadata
I20250114 02:55:15.114519 3961 raft_consensus.cc:357] T a02695476ce44eea9bc7a53871bc45a0 P 7b7de0f9038c4ba287750ad132cb3597 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:15.122181 3960 tablet_bootstrap.cc:492] T a02695476ce44eea9bc7a53871bc45a0 P 5017c179ddc640fc9ebf5dea7f3957f8: No bootstrap required, opened a new log
I20250114 02:55:15.122687 3960 ts_tablet_manager.cc:1397] T a02695476ce44eea9bc7a53871bc45a0 P 5017c179ddc640fc9ebf5dea7f3957f8: Time spent bootstrapping tablet: real 0.043s user 0.007s sys 0.007s
I20250114 02:55:15.129204 3961 raft_consensus.cc:383] T a02695476ce44eea9bc7a53871bc45a0 P 7b7de0f9038c4ba287750ad132cb3597 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:55:15.129971 3957 ts_tablet_manager.cc:1428] T 83abece59e0541f1ad0eefee1a8e7994 P 39c64942315c4716baaee0c251c05acd: Time spent starting tablet: real 0.127s user 0.039s sys 0.054s
I20250114 02:55:15.130532 3537 heartbeater.cc:502] Master 127.0.107.125:38653 was elected leader, sending a full tablet report...
I20250114 02:55:15.130872 3957 tablet_bootstrap.cc:492] T 7441da6add1a46e7b18148ea27d05323 P 39c64942315c4716baaee0c251c05acd: Bootstrap starting.
I20250114 02:55:15.138203 3957 tablet_bootstrap.cc:654] T 7441da6add1a46e7b18148ea27d05323 P 39c64942315c4716baaee0c251c05acd: Neither blocks nor log segments found. Creating new log.
I20250114 02:55:15.140534 3961 raft_consensus.cc:738] T a02695476ce44eea9bc7a53871bc45a0 P 7b7de0f9038c4ba287750ad132cb3597 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b7de0f9038c4ba287750ad132cb3597, State: Initialized, Role: FOLLOWER
I20250114 02:55:15.125280 3960 raft_consensus.cc:357] T a02695476ce44eea9bc7a53871bc45a0 P 5017c179ddc640fc9ebf5dea7f3957f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:15.149318 3961 consensus_queue.cc:260] T a02695476ce44eea9bc7a53871bc45a0 P 7b7de0f9038c4ba287750ad132cb3597 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:15.150297 3957 tablet_bootstrap.cc:492] T 7441da6add1a46e7b18148ea27d05323 P 39c64942315c4716baaee0c251c05acd: No bootstrap required, opened a new log
I20250114 02:55:15.150262 3960 raft_consensus.cc:383] T a02695476ce44eea9bc7a53871bc45a0 P 5017c179ddc640fc9ebf5dea7f3957f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:55:15.150633 3960 raft_consensus.cc:738] T a02695476ce44eea9bc7a53871bc45a0 P 5017c179ddc640fc9ebf5dea7f3957f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5017c179ddc640fc9ebf5dea7f3957f8, State: Initialized, Role: FOLLOWER
I20250114 02:55:15.150748 3957 ts_tablet_manager.cc:1397] T 7441da6add1a46e7b18148ea27d05323 P 39c64942315c4716baaee0c251c05acd: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.008s
I20250114 02:55:15.151533 3960 consensus_queue.cc:260] T a02695476ce44eea9bc7a53871bc45a0 P 5017c179ddc640fc9ebf5dea7f3957f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:15.155624 3957 raft_consensus.cc:357] T 7441da6add1a46e7b18148ea27d05323 P 39c64942315c4716baaee0c251c05acd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:15.156384 3957 raft_consensus.cc:383] T 7441da6add1a46e7b18148ea27d05323 P 39c64942315c4716baaee0c251c05acd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:55:15.156802 3957 raft_consensus.cc:738] T 7441da6add1a46e7b18148ea27d05323 P 39c64942315c4716baaee0c251c05acd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39c64942315c4716baaee0c251c05acd, State: Initialized, Role: FOLLOWER
I20250114 02:55:15.157521 3957 consensus_queue.cc:260] T 7441da6add1a46e7b18148ea27d05323 P 39c64942315c4716baaee0c251c05acd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:15.160701 3957 ts_tablet_manager.cc:1428] T 7441da6add1a46e7b18148ea27d05323 P 39c64942315c4716baaee0c251c05acd: Time spent starting tablet: real 0.010s user 0.001s sys 0.005s
I20250114 02:55:15.161585 3957 tablet_bootstrap.cc:492] T a02695476ce44eea9bc7a53871bc45a0 P 39c64942315c4716baaee0c251c05acd: Bootstrap starting.
I20250114 02:55:15.165362 3961 ts_tablet_manager.cc:1428] T a02695476ce44eea9bc7a53871bc45a0 P 7b7de0f9038c4ba287750ad132cb3597: Time spent starting tablet: real 0.066s user 0.009s sys 0.000s
I20250114 02:55:15.168877 3957 tablet_bootstrap.cc:654] T a02695476ce44eea9bc7a53871bc45a0 P 39c64942315c4716baaee0c251c05acd: Neither blocks nor log segments found. Creating new log.
I20250114 02:55:15.181711 3957 tablet_bootstrap.cc:492] T a02695476ce44eea9bc7a53871bc45a0 P 39c64942315c4716baaee0c251c05acd: No bootstrap required, opened a new log
I20250114 02:55:15.182241 3957 ts_tablet_manager.cc:1397] T a02695476ce44eea9bc7a53871bc45a0 P 39c64942315c4716baaee0c251c05acd: Time spent bootstrapping tablet: real 0.021s user 0.012s sys 0.005s
I20250114 02:55:15.185083 3960 ts_tablet_manager.cc:1428] T a02695476ce44eea9bc7a53871bc45a0 P 5017c179ddc640fc9ebf5dea7f3957f8: Time spent starting tablet: real 0.062s user 0.006s sys 0.000s
I20250114 02:55:15.184823 3957 raft_consensus.cc:357] T a02695476ce44eea9bc7a53871bc45a0 P 39c64942315c4716baaee0c251c05acd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:15.185771 3957 raft_consensus.cc:383] T a02695476ce44eea9bc7a53871bc45a0 P 39c64942315c4716baaee0c251c05acd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:55:15.186092 3957 raft_consensus.cc:738] T a02695476ce44eea9bc7a53871bc45a0 P 39c64942315c4716baaee0c251c05acd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39c64942315c4716baaee0c251c05acd, State: Initialized, Role: FOLLOWER
I20250114 02:55:15.187877 3957 consensus_queue.cc:260] T a02695476ce44eea9bc7a53871bc45a0 P 39c64942315c4716baaee0c251c05acd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:15.190588 3957 ts_tablet_manager.cc:1428] T a02695476ce44eea9bc7a53871bc45a0 P 39c64942315c4716baaee0c251c05acd: Time spent starting tablet: real 0.008s user 0.003s sys 0.002s
I20250114 02:55:15.195390 3961 tablet_bootstrap.cc:492] T 7441da6add1a46e7b18148ea27d05323 P 7b7de0f9038c4ba287750ad132cb3597: Bootstrap starting.
I20250114 02:55:15.202589 3961 tablet_bootstrap.cc:654] T 7441da6add1a46e7b18148ea27d05323 P 7b7de0f9038c4ba287750ad132cb3597: Neither blocks nor log segments found. Creating new log.
I20250114 02:55:15.213788 3955 ts_tablet_manager.cc:1918] T 5e5a4f58a4ba47e59477208eb2bd833b P 39c64942315c4716baaee0c251c05acd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250114 02:55:15.214284 3955 log.cc:1198] T 5e5a4f58a4ba47e59477208eb2bd833b P 39c64942315c4716baaee0c251c05acd: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/5e5a4f58a4ba47e59477208eb2bd833b
I20250114 02:55:15.215025 3955 ts_tablet_manager.cc:1939] T 5e5a4f58a4ba47e59477208eb2bd833b P 39c64942315c4716baaee0c251c05acd: Deleting consensus metadata
I20250114 02:55:15.217391 3955 tablet_replica.cc:331] T 83abece59e0541f1ad0eefee1a8e7994 P 39c64942315c4716baaee0c251c05acd: stopping tablet replica
I20250114 02:55:15.218113 3955 raft_consensus.cc:2238] T 83abece59e0541f1ad0eefee1a8e7994 P 39c64942315c4716baaee0c251c05acd [term 0 FOLLOWER]: Raft consensus shutting down.
I20250114 02:55:15.218596 3955 raft_consensus.cc:2267] T 83abece59e0541f1ad0eefee1a8e7994 P 39c64942315c4716baaee0c251c05acd [term 0 FOLLOWER]: Raft consensus is shut down!
I20250114 02:55:15.221423 3955 ts_tablet_manager.cc:1905] T 83abece59e0541f1ad0eefee1a8e7994 P 39c64942315c4716baaee0c251c05acd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250114 02:55:15.235975 3955 ts_tablet_manager.cc:1918] T 83abece59e0541f1ad0eefee1a8e7994 P 39c64942315c4716baaee0c251c05acd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250114 02:55:15.236452 3955 log.cc:1198] T 83abece59e0541f1ad0eefee1a8e7994 P 39c64942315c4716baaee0c251c05acd: Deleting WAL directory at /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/ts-1/wal/wals/83abece59e0541f1ad0eefee1a8e7994
I20250114 02:55:15.237131 3955 ts_tablet_manager.cc:1939] T 83abece59e0541f1ad0eefee1a8e7994 P 39c64942315c4716baaee0c251c05acd: Deleting consensus metadata
I20250114 02:55:15.242064 3961 tablet_bootstrap.cc:492] T 7441da6add1a46e7b18148ea27d05323 P 7b7de0f9038c4ba287750ad132cb3597: No bootstrap required, opened a new log
I20250114 02:55:15.242544 3961 ts_tablet_manager.cc:1397] T 7441da6add1a46e7b18148ea27d05323 P 7b7de0f9038c4ba287750ad132cb3597: Time spent bootstrapping tablet: real 0.047s user 0.007s sys 0.004s
I20250114 02:55:15.244967 3961 raft_consensus.cc:357] T 7441da6add1a46e7b18148ea27d05323 P 7b7de0f9038c4ba287750ad132cb3597 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:15.252542 3961 raft_consensus.cc:383] T 7441da6add1a46e7b18148ea27d05323 P 7b7de0f9038c4ba287750ad132cb3597 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 02:55:15.252878 3961 raft_consensus.cc:738] T 7441da6add1a46e7b18148ea27d05323 P 7b7de0f9038c4ba287750ad132cb3597 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b7de0f9038c4ba287750ad132cb3597, State: Initialized, Role: FOLLOWER
I20250114 02:55:15.253571 3961 consensus_queue.cc:260] T 7441da6add1a46e7b18148ea27d05323 P 7b7de0f9038c4ba287750ad132cb3597 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:15.255853 3961 ts_tablet_manager.cc:1428] T 7441da6add1a46e7b18148ea27d05323 P 7b7de0f9038c4ba287750ad132cb3597: Time spent starting tablet: real 0.013s user 0.004s sys 0.000s
I20250114 02:55:15.334223 3975 raft_consensus.cc:491] T 7441da6add1a46e7b18148ea27d05323 P 39c64942315c4716baaee0c251c05acd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:55:15.334751 3975 raft_consensus.cc:513] T 7441da6add1a46e7b18148ea27d05323 P 39c64942315c4716baaee0c251c05acd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:15.336759 3975 leader_election.cc:290] T 7441da6add1a46e7b18148ea27d05323 P 39c64942315c4716baaee0c251c05acd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b7de0f9038c4ba287750ad132cb3597 (127.0.107.65:34329), 5017c179ddc640fc9ebf5dea7f3957f8 (127.0.107.67:45409)
I20250114 02:55:15.338790 3633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7441da6add1a46e7b18148ea27d05323" candidate_uuid: "39c64942315c4716baaee0c251c05acd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" is_pre_election: true
I20250114 02:55:15.339388 3633 raft_consensus.cc:2463] T 7441da6add1a46e7b18148ea27d05323 P 5017c179ddc640fc9ebf5dea7f3957f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39c64942315c4716baaee0c251c05acd in term 0.
I20250114 02:55:15.341229 3346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7441da6add1a46e7b18148ea27d05323" candidate_uuid: "39c64942315c4716baaee0c251c05acd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b7de0f9038c4ba287750ad132cb3597" is_pre_election: true
I20250114 02:55:15.341974 3346 raft_consensus.cc:2463] T 7441da6add1a46e7b18148ea27d05323 P 7b7de0f9038c4ba287750ad132cb3597 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39c64942315c4716baaee0c251c05acd in term 0.
I20250114 02:55:15.343336 3423 leader_election.cc:304] T 7441da6add1a46e7b18148ea27d05323 P 39c64942315c4716baaee0c251c05acd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39c64942315c4716baaee0c251c05acd, 5017c179ddc640fc9ebf5dea7f3957f8; no voters:
I20250114 02:55:15.345717 3975 raft_consensus.cc:2798] T 7441da6add1a46e7b18148ea27d05323 P 39c64942315c4716baaee0c251c05acd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:55:15.346146 3975 raft_consensus.cc:491] T 7441da6add1a46e7b18148ea27d05323 P 39c64942315c4716baaee0c251c05acd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:55:15.346644 3975 raft_consensus.cc:3054] T 7441da6add1a46e7b18148ea27d05323 P 39c64942315c4716baaee0c251c05acd [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:55:15.356093 3975 raft_consensus.cc:513] T 7441da6add1a46e7b18148ea27d05323 P 39c64942315c4716baaee0c251c05acd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:15.359287 3346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7441da6add1a46e7b18148ea27d05323" candidate_uuid: "39c64942315c4716baaee0c251c05acd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b7de0f9038c4ba287750ad132cb3597"
I20250114 02:55:15.359840 3346 raft_consensus.cc:3054] T 7441da6add1a46e7b18148ea27d05323 P 7b7de0f9038c4ba287750ad132cb3597 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:55:15.359845 3975 leader_election.cc:290] T 7441da6add1a46e7b18148ea27d05323 P 39c64942315c4716baaee0c251c05acd [CANDIDATE]: Term 1 election: Requested vote from peers 7b7de0f9038c4ba287750ad132cb3597 (127.0.107.65:34329), 5017c179ddc640fc9ebf5dea7f3957f8 (127.0.107.67:45409)
I20250114 02:55:15.361562 3633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7441da6add1a46e7b18148ea27d05323" candidate_uuid: "39c64942315c4716baaee0c251c05acd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5017c179ddc640fc9ebf5dea7f3957f8"
I20250114 02:55:15.362145 3633 raft_consensus.cc:3054] T 7441da6add1a46e7b18148ea27d05323 P 5017c179ddc640fc9ebf5dea7f3957f8 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:55:15.368070 3346 raft_consensus.cc:2463] T 7441da6add1a46e7b18148ea27d05323 P 7b7de0f9038c4ba287750ad132cb3597 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39c64942315c4716baaee0c251c05acd in term 1.
I20250114 02:55:15.369626 3633 raft_consensus.cc:2463] T 7441da6add1a46e7b18148ea27d05323 P 5017c179ddc640fc9ebf5dea7f3957f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39c64942315c4716baaee0c251c05acd in term 1.
I20250114 02:55:15.369478 3423 leader_election.cc:304] T 7441da6add1a46e7b18148ea27d05323 P 39c64942315c4716baaee0c251c05acd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39c64942315c4716baaee0c251c05acd, 7b7de0f9038c4ba287750ad132cb3597; no voters:
I20250114 02:55:15.370620 3975 raft_consensus.cc:2798] T 7441da6add1a46e7b18148ea27d05323 P 39c64942315c4716baaee0c251c05acd [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:55:15.371641 3975 raft_consensus.cc:695] T 7441da6add1a46e7b18148ea27d05323 P 39c64942315c4716baaee0c251c05acd [term 1 LEADER]: Becoming Leader. State: Replica: 39c64942315c4716baaee0c251c05acd, State: Running, Role: LEADER
I20250114 02:55:15.372561 3975 consensus_queue.cc:237] T 7441da6add1a46e7b18148ea27d05323 P 39c64942315c4716baaee0c251c05acd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
W20250114 02:55:15.403798 3283 connection.cc:537] server connection from 127.0.0.1:58840 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250114 02:55:15.403759 3566 connection.cc:537] server connection from 127.0.0.1:60676 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250114 02:55:15.403935 3690 connection.cc:537] client connection to 127.0.107.125:38653 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250114 02:55:15.404621 3425 connection.cc:537] client connection to 127.0.107.125:38653 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250114 02:55:15.405861 3690 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.0.107.125:38653)
W20250114 02:55:15.406574 3690 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.107.125:38653)
W20250114 02:55:15.406884 3423 connection.cc:537] server connection from 127.0.0.1:58624 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250114 02:55:15.407230 3690 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.107.125:38653)
W20250114 02:55:15.407548 3282 connection.cc:537] client connection to 127.0.107.125:38653 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250114 02:55:15.407606 3568 connection.cc:537] client connection to 127.0.107.125:38653 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250114 02:55:15.416262 3690 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.0.107.125:38653)
I20250114 02:55:15.498632 3987 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bfb2e2fecfb94af8bf892c6f774c2d89)
I20250114 02:55:15.499219 3987 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:55:15.502261 3987 leader_election.cc:290] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235), bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653)
W20250114 02:55:15.505513 3858 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111)
W20250114 02:55:15.510385 3858 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235): Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111)
W20250114 02:55:15.510897 3858 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653): Network error: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)
I20250114 02:55:15.511193 3858 leader_election.cc:304] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9a6e2eab6e541eca12ee79c1d3004e8; no voters: a8f3d9567c014741a7e628bab3e18e27, bfb2e2fecfb94af8bf892c6f774c2d89
I20250114 02:55:15.511795 3987 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:55:15.520967 3937 raft_consensus.cc:491] T a02695476ce44eea9bc7a53871bc45a0 P 7b7de0f9038c4ba287750ad132cb3597 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:55:15.521363 3937 raft_consensus.cc:513] T a02695476ce44eea9bc7a53871bc45a0 P 7b7de0f9038c4ba287750ad132cb3597 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:15.522904 3937 leader_election.cc:290] T a02695476ce44eea9bc7a53871bc45a0 P 7b7de0f9038c4ba287750ad132cb3597 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39c64942315c4716baaee0c251c05acd (127.0.107.66:33487), 5017c179ddc640fc9ebf5dea7f3957f8 (127.0.107.67:45409)
I20250114 02:55:15.523821 3490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a02695476ce44eea9bc7a53871bc45a0" candidate_uuid: "7b7de0f9038c4ba287750ad132cb3597" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39c64942315c4716baaee0c251c05acd" is_pre_election: true
I20250114 02:55:15.524078 3633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a02695476ce44eea9bc7a53871bc45a0" candidate_uuid: "7b7de0f9038c4ba287750ad132cb3597" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" is_pre_election: true
I20250114 02:55:15.524369 3490 raft_consensus.cc:2463] T a02695476ce44eea9bc7a53871bc45a0 P 39c64942315c4716baaee0c251c05acd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b7de0f9038c4ba287750ad132cb3597 in term 0.
I20250114 02:55:15.524739 3633 raft_consensus.cc:2463] T a02695476ce44eea9bc7a53871bc45a0 P 5017c179ddc640fc9ebf5dea7f3957f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b7de0f9038c4ba287750ad132cb3597 in term 0.
I20250114 02:55:15.525385 3281 leader_election.cc:304] T a02695476ce44eea9bc7a53871bc45a0 P 7b7de0f9038c4ba287750ad132cb3597 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39c64942315c4716baaee0c251c05acd, 7b7de0f9038c4ba287750ad132cb3597; no voters:
I20250114 02:55:15.526006 3937 raft_consensus.cc:2798] T a02695476ce44eea9bc7a53871bc45a0 P 7b7de0f9038c4ba287750ad132cb3597 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 02:55:15.526257 3937 raft_consensus.cc:491] T a02695476ce44eea9bc7a53871bc45a0 P 7b7de0f9038c4ba287750ad132cb3597 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 02:55:15.526477 3937 raft_consensus.cc:3054] T a02695476ce44eea9bc7a53871bc45a0 P 7b7de0f9038c4ba287750ad132cb3597 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:55:15.530720 3937 raft_consensus.cc:513] T a02695476ce44eea9bc7a53871bc45a0 P 7b7de0f9038c4ba287750ad132cb3597 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:15.532936 3490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a02695476ce44eea9bc7a53871bc45a0" candidate_uuid: "7b7de0f9038c4ba287750ad132cb3597" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39c64942315c4716baaee0c251c05acd"
I20250114 02:55:15.533079 3633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a02695476ce44eea9bc7a53871bc45a0" candidate_uuid: "7b7de0f9038c4ba287750ad132cb3597" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5017c179ddc640fc9ebf5dea7f3957f8"
I20250114 02:55:15.533488 3490 raft_consensus.cc:3054] T a02695476ce44eea9bc7a53871bc45a0 P 39c64942315c4716baaee0c251c05acd [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:55:15.533613 3633 raft_consensus.cc:3054] T a02695476ce44eea9bc7a53871bc45a0 P 5017c179ddc640fc9ebf5dea7f3957f8 [term 0 FOLLOWER]: Advancing to term 1
I20250114 02:55:15.532107 3937 leader_election.cc:290] T a02695476ce44eea9bc7a53871bc45a0 P 7b7de0f9038c4ba287750ad132cb3597 [CANDIDATE]: Term 1 election: Requested vote from peers 39c64942315c4716baaee0c251c05acd (127.0.107.66:33487), 5017c179ddc640fc9ebf5dea7f3957f8 (127.0.107.67:45409)
I20250114 02:55:15.540621 3633 raft_consensus.cc:2463] T a02695476ce44eea9bc7a53871bc45a0 P 5017c179ddc640fc9ebf5dea7f3957f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b7de0f9038c4ba287750ad132cb3597 in term 1.
I20250114 02:55:15.539952 3490 raft_consensus.cc:2463] T a02695476ce44eea9bc7a53871bc45a0 P 39c64942315c4716baaee0c251c05acd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b7de0f9038c4ba287750ad132cb3597 in term 1.
I20250114 02:55:15.541661 3280 leader_election.cc:304] T a02695476ce44eea9bc7a53871bc45a0 P 7b7de0f9038c4ba287750ad132cb3597 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5017c179ddc640fc9ebf5dea7f3957f8, 7b7de0f9038c4ba287750ad132cb3597; no voters:
I20250114 02:55:15.542351 3937 raft_consensus.cc:2798] T a02695476ce44eea9bc7a53871bc45a0 P 7b7de0f9038c4ba287750ad132cb3597 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 02:55:15.543365 3937 raft_consensus.cc:695] T a02695476ce44eea9bc7a53871bc45a0 P 7b7de0f9038c4ba287750ad132cb3597 [term 1 LEADER]: Becoming Leader. State: Replica: 7b7de0f9038c4ba287750ad132cb3597, State: Running, Role: LEADER
I20250114 02:55:15.544256 3937 consensus_queue.cc:237] T a02695476ce44eea9bc7a53871bc45a0 P 7b7de0f9038c4ba287750ad132cb3597 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 } } peers { permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 } } peers { permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 } }
I20250114 02:55:16.251053 4000 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bfb2e2fecfb94af8bf892c6f774c2d89)
I20250114 02:55:16.251432 4000 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:55:16.252907 4000 leader_election.cc:290] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235), bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653)
W20250114 02:55:16.259869 3858 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235): Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111)
W20250114 02:55:16.260509 3858 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653): Network error: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)
I20250114 02:55:16.260957 3858 leader_election.cc:304] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9a6e2eab6e541eca12ee79c1d3004e8; no voters: a8f3d9567c014741a7e628bab3e18e27, bfb2e2fecfb94af8bf892c6f774c2d89
I20250114 02:55:16.261777 4000 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:55:16.345371 4004 consensus_queue.cc:1035] T 7441da6add1a46e7b18148ea27d05323 P 39c64942315c4716baaee0c251c05acd [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" member_type: VOTER last_known_addr { host: "127.0.107.65" port: 34329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:55:16.365322 4004 consensus_queue.cc:1035] T 7441da6add1a46e7b18148ea27d05323 P 39c64942315c4716baaee0c251c05acd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:55:16.402225 429 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrCUXNq/build/tsan/bin/kudu with pid 3751
W20250114 02:55:16.436630 3691 connection.cc:537] client connection to 127.0.107.124:44431 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250114 02:55:16.702862 4005 consensus_queue.cc:1035] T a02695476ce44eea9bc7a53871bc45a0 P 7b7de0f9038c4ba287750ad132cb3597 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39c64942315c4716baaee0c251c05acd" member_type: VOTER last_known_addr { host: "127.0.107.66" port: 33487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:55:16.717381 4005 consensus_queue.cc:1035] T a02695476ce44eea9bc7a53871bc45a0 P 7b7de0f9038c4ba287750ad132cb3597 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" member_type: VOTER last_known_addr { host: "127.0.107.67" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 02:55:17.346297 429 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrCUXNq/build/tsan/bin/kudu
/tmp/dist-test-taskrCUXNq/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.107.124:44431
--webserver_interface=127.0.107.124
--webserver_port=36967
--builtin_ntp_servers=127.0.107.84:38241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.107.126:40235,127.0.107.125:38653,127.0.107.124:44431
--hive_metastore_uris=thrift://127.0.0.1:33943
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250114 02:55:17.689834 4026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 02:55:17.690407 4026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 02:55:17.690810 4026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 02:55:17.724776 4026 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20250114 02:55:17.725021 4026 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20250114 02:55:17.725224 4026 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250114 02:55:17.725474 4026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 02:55:17.725809 4026 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250114 02:55:17.726114 4026 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250114 02:55:17.763190 4026 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.107.84:38241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:33943
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.107.126:40235,127.0.107.125:38653,127.0.107.124:44431
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.107.124:44431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.107.124
--webserver_port=36967
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type FASTDEBUG
built by None at 14 Jan 2025 02:43:23 UTC on a633016ff5c9
build id 4168
TSAN enabled
I20250114 02:55:17.764642 4026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 02:55:17.766264 4026 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 02:55:17.783140 4041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 02:55:17.783792 4042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 02:55:17.785619 4026 server_base.cc:1034] running on GCE node
W20250114 02:55:17.784857 4044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 02:55:19.007663 4026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 02:55:19.010929 4026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 02:55:19.012413 4026 hybrid_clock.cc:648] HybridClock initialized: now 1736823319012361 us; error 65 us; skew 500 ppm
I20250114 02:55:19.013386 4026 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250114 02:55:19.020112 4026 webserver.cc:458] Webserver started at http://127.0.107.124:36967/ using document root <none> and password file <none>
I20250114 02:55:19.021096 4026 fs_manager.cc:362] Metadata directory not provided
I20250114 02:55:19.021308 4026 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 02:55:19.029537 4026 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.008s sys 0.000s
I20250114 02:55:19.034503 4072 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 02:55:19.035580 4026 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250114 02:55:19.035888 4026 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-2/data,/tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-2/wal
uuid: "f9a6e2eab6e541eca12ee79c1d3004e8"
format_stamp: "Formatted at 2025-01-14 02:55:01 on dist-test-slave-36vg"
I20250114 02:55:19.038319 4026 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 02:55:19.102524 4026 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 02:55:19.104351 4026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 02:55:19.104972 4026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 02:55:19.175629 4026 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.124:44431
I20250114 02:55:19.175716 4124 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.124:44431 every 8 connection(s)
I20250114 02:55:19.178493 4026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrCUXNq/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736823234046404-429-0/minicluster-data/master-2/data/info.pb
I20250114 02:55:19.180567 429 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrCUXNq/build/tsan/bin/kudu as pid 4026
I20250114 02:55:19.192281 4125 sys_catalog.cc:263] Verifying existing consensus state
I20250114 02:55:19.199220 4125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8: Bootstrap starting.
I20250114 02:55:19.218461 4125 log.cc:826] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8: Log is configured to *not* fsync() on all Append() calls
I20250114 02:55:19.280443 3678 heartbeater.cc:346] Connected to a master server at 127.0.107.124:44431
I20250114 02:55:19.287791 4125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=16 ignored=0} mutations{seen=21 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250114 02:55:19.288551 4125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8: Bootstrap complete.
I20250114 02:55:19.305502 4125 raft_consensus.cc:357] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:55:19.308071 4125 raft_consensus.cc:738] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9a6e2eab6e541eca12ee79c1d3004e8, State: Initialized, Role: FOLLOWER
I20250114 02:55:19.308741 4125 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 3.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:55:19.311443 4133 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } } }
I20250114 02:55:19.312206 4133 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [sys.catalog]: This master's current role is: FOLLOWER
I20250114 02:55:19.312515 4125 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [sys.catalog]: configured and running, proceeding with master startup.
I20250114 02:55:19.342895 4148 catalog_manager.cc:1260] Loaded cluster ID: 383d09ac0a7b47d1bf47222602c9cf34
I20250114 02:55:19.343279 4148 catalog_manager.cc:1553] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8: loading cluster ID for follower catalog manager: success
I20250114 02:55:19.351305 4148 catalog_manager.cc:1575] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8: acquiring CA information for follower catalog manager: success
I20250114 02:55:19.357486 4148 catalog_manager.cc:1603] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250114 02:55:19.338436 4026 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 02:55:19.372750 4026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250114 02:55:19.413439 3536 heartbeater.cc:346] Connected to a master server at 127.0.107.124:44431
I20250114 02:55:19.419085 4090 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "39c64942315c4716baaee0c251c05acd" instance_seqno: 1736823307893419) as {username='slave'} at 127.0.107.66:40401; Asking this server to re-register.
I20250114 02:55:19.420403 3536 heartbeater.cc:463] Registering TS with master...
I20250114 02:55:19.423316 4090 ts_manager.cc:194] Registered new tserver with Master: 39c64942315c4716baaee0c251c05acd (127.0.107.66:33487)
I20250114 02:55:19.629455 3392 heartbeater.cc:346] Connected to a master server at 127.0.107.124:44431
I20250114 02:55:19.632227 4090 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "7b7de0f9038c4ba287750ad132cb3597" instance_seqno: 1736823305980627) as {username='slave'} at 127.0.107.65:47223; Asking this server to re-register.
I20250114 02:55:19.633253 3392 heartbeater.cc:463] Registering TS with master...
I20250114 02:55:19.635749 4090 ts_manager.cc:194] Registered new tserver with Master: 7b7de0f9038c4ba287750ad132cb3597 (127.0.107.65:34329)
I20250114 02:55:19.788516 4090 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "5017c179ddc640fc9ebf5dea7f3957f8" instance_seqno: 1736823309820499) as {username='slave'} at 127.0.107.67:35817; Asking this server to re-register.
I20250114 02:55:19.789744 3678 heartbeater.cc:463] Registering TS with master...
I20250114 02:55:19.792589 4090 ts_manager.cc:194] Registered new tserver with Master: 5017c179ddc640fc9ebf5dea7f3957f8 (127.0.107.67:45409)
I20250114 02:55:19.930008 4156 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:55:19.930550 4156 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:55:19.934347 4156 leader_election.cc:290] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235), bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653)
W20250114 02:55:19.938040 4075 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111)
W20250114 02:55:19.942536 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235): Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111)
W20250114 02:55:19.943274 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653): Network error: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)
I20250114 02:55:19.943668 4075 leader_election.cc:304] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9a6e2eab6e541eca12ee79c1d3004e8; no voters: a8f3d9567c014741a7e628bab3e18e27, bfb2e2fecfb94af8bf892c6f774c2d89
I20250114 02:55:19.944514 4156 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:55:20.514421 4167 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:55:20.514876 4167 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:55:20.516786 4167 leader_election.cc:290] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235), bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653)
W20250114 02:55:20.524163 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235): Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111)
W20250114 02:55:20.524863 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653): Network error: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)
I20250114 02:55:20.525170 4075 leader_election.cc:304] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9a6e2eab6e541eca12ee79c1d3004e8; no voters: a8f3d9567c014741a7e628bab3e18e27, bfb2e2fecfb94af8bf892c6f774c2d89
I20250114 02:55:20.525734 4167 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:55:21.027187 4179 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:55:21.027580 4179 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:55:21.029284 4179 leader_election.cc:290] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235), bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653)
W20250114 02:55:21.034116 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235): Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111)
W20250114 02:55:21.034674 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653): Network error: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)
I20250114 02:55:21.034940 4075 leader_election.cc:304] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9a6e2eab6e541eca12ee79c1d3004e8; no voters: a8f3d9567c014741a7e628bab3e18e27, bfb2e2fecfb94af8bf892c6f774c2d89
I20250114 02:55:21.035490 4179 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:55:21.628067 429 master-stress-test.cc:577] Killing test thread 1/10
I20250114 02:55:22.163707 4204 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:55:22.164069 4204 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:55:22.165496 4204 leader_election.cc:290] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235), bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653)
W20250114 02:55:22.170254 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235): Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111)
W20250114 02:55:22.170929 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653): Network error: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)
I20250114 02:55:22.171224 4075 leader_election.cc:304] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9a6e2eab6e541eca12ee79c1d3004e8; no voters: a8f3d9567c014741a7e628bab3e18e27, bfb2e2fecfb94af8bf892c6f774c2d89
I20250114 02:55:22.171738 4204 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250114 02:55:22.904752 3567 consensus_peers.cc:487] T c174bcbc5d51402b8ae332b87ffa2139 P 5017c179ddc640fc9ebf5dea7f3957f8 -> Peer 39c64942315c4716baaee0c251c05acd (127.0.107.66:33487): Couldn't send request to peer 39c64942315c4716baaee0c251c05acd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c174bcbc5d51402b8ae332b87ffa2139. This is attempt 6: this message will repeat every 5th retry.
I20250114 02:55:23.468080 4227 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:55:23.468504 4227 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:55:23.469959 4227 leader_election.cc:290] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235), bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653)
W20250114 02:55:23.474735 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235): Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111)
W20250114 02:55:23.475401 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653): Network error: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)
I20250114 02:55:23.475677 4075 leader_election.cc:304] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9a6e2eab6e541eca12ee79c1d3004e8; no voters: a8f3d9567c014741a7e628bab3e18e27, bfb2e2fecfb94af8bf892c6f774c2d89
I20250114 02:55:23.476197 4227 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:55:24.375461 4026 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 02:55:24.375874 4026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250114 02:55:25.176414 4262 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:55:25.176846 4262 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:55:25.178517 4262 leader_election.cc:290] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235), bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653)
W20250114 02:55:25.180645 4075 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20250114 02:55:25.183772 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235): Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111)
W20250114 02:55:25.184396 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653): Network error: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)
I20250114 02:55:25.184656 4075 leader_election.cc:304] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9a6e2eab6e541eca12ee79c1d3004e8; no voters: a8f3d9567c014741a7e628bab3e18e27, bfb2e2fecfb94af8bf892c6f774c2d89
I20250114 02:55:25.185192 4262 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:55:28.166343 4314 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:55:28.166955 4314 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:55:28.168987 4314 leader_election.cc:290] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235), bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653)
W20250114 02:55:28.198148 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235): Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111)
W20250114 02:55:28.199021 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653): Network error: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)
I20250114 02:55:28.199537 4075 leader_election.cc:304] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9a6e2eab6e541eca12ee79c1d3004e8; no voters: a8f3d9567c014741a7e628bab3e18e27, bfb2e2fecfb94af8bf892c6f774c2d89
I20250114 02:55:28.200601 4314 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:55:29.378731 4026 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 02:55:29.379173 4026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250114 02:55:31.064739 4364 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:55:31.065114 4364 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:55:31.066548 4364 leader_election.cc:290] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235), bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653)
W20250114 02:55:31.068687 4075 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250114 02:55:31.071087 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235): Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111)
W20250114 02:55:31.071825 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653): Network error: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)
I20250114 02:55:31.072089 4075 leader_election.cc:304] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9a6e2eab6e541eca12ee79c1d3004e8; no voters: a8f3d9567c014741a7e628bab3e18e27, bfb2e2fecfb94af8bf892c6f774c2d89
I20250114 02:55:31.072624 4364 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250114 02:55:32.927924 3567 consensus_peers.cc:487] T c174bcbc5d51402b8ae332b87ffa2139 P 5017c179ddc640fc9ebf5dea7f3957f8 -> Peer 39c64942315c4716baaee0c251c05acd (127.0.107.66:33487): Couldn't send request to peer 39c64942315c4716baaee0c251c05acd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c174bcbc5d51402b8ae332b87ffa2139. This is attempt 11: this message will repeat every 5th retry.
I20250114 02:55:34.381925 4026 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 02:55:34.382301 4026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250114 02:55:39.385628 4026 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 02:55:39.386035 4026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250114 02:55:42.271406 3567 consensus_peers.cc:487] T c174bcbc5d51402b8ae332b87ffa2139 P 5017c179ddc640fc9ebf5dea7f3957f8 -> Peer 39c64942315c4716baaee0c251c05acd (127.0.107.66:33487): Couldn't send request to peer 39c64942315c4716baaee0c251c05acd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c174bcbc5d51402b8ae332b87ffa2139. This is attempt 16: this message will repeat every 5th retry.
I20250114 02:55:42.482026 4564 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:55:42.482441 4564 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:55:42.483870 4564 leader_election.cc:290] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235), bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653)
W20250114 02:55:42.486713 4075 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250114 02:55:42.490486 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235): Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111)
W20250114 02:55:42.491256 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653): Network error: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)
I20250114 02:55:42.491551 4075 leader_election.cc:304] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9a6e2eab6e541eca12ee79c1d3004e8; no voters: a8f3d9567c014741a7e628bab3e18e27, bfb2e2fecfb94af8bf892c6f774c2d89
I20250114 02:55:42.492251 4564 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:55:44.388741 4026 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 02:55:44.389088 4026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250114 02:55:44.695956 3532 debug-util.cc:398] Leaking SignalData structure 0x7b0800072560 after lost signal to thread 3403
W20250114 02:55:44.696911 3532 debug-util.cc:398] Leaking SignalData structure 0x7b0800044a80 after lost signal to thread 3535
I20250114 02:55:49.392815 4026 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 02:55:49.393226 4026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250114 02:55:52.345499 3567 consensus_peers.cc:487] T c174bcbc5d51402b8ae332b87ffa2139 P 5017c179ddc640fc9ebf5dea7f3957f8 -> Peer 39c64942315c4716baaee0c251c05acd (127.0.107.66:33487): Couldn't send request to peer 39c64942315c4716baaee0c251c05acd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c174bcbc5d51402b8ae332b87ffa2139. This is attempt 21: this message will repeat every 5th retry.
I20250114 02:55:53.849679 4756 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:55:53.850239 4756 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:55:53.852061 4756 leader_election.cc:290] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235), bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653)
W20250114 02:55:53.854106 4075 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250114 02:55:53.856678 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235): Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111)
W20250114 02:55:53.858238 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653): Network error: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)
I20250114 02:55:53.858505 4075 leader_election.cc:304] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9a6e2eab6e541eca12ee79c1d3004e8; no voters: a8f3d9567c014741a7e628bab3e18e27, bfb2e2fecfb94af8bf892c6f774c2d89
I20250114 02:55:53.859014 4756 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:55:54.395792 4026 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 02:55:54.396138 4026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250114 02:55:59.400172 4026 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 02:55:59.400625 4026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250114 02:56:01.654351 3567 consensus_peers.cc:487] T c174bcbc5d51402b8ae332b87ffa2139 P 5017c179ddc640fc9ebf5dea7f3957f8 -> Peer 39c64942315c4716baaee0c251c05acd (127.0.107.66:33487): Couldn't send request to peer 39c64942315c4716baaee0c251c05acd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c174bcbc5d51402b8ae332b87ffa2139. This is attempt 26: this message will repeat every 5th retry.
I20250114 02:56:04.403472 4026 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 02:56:04.403872 4026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250114 02:56:09.407215 4026 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 02:56:09.407677 4026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250114 02:56:10.479147 3393 heartbeater.cc:643] Failed to heartbeat to 127.0.107.125:38653 (115 consecutive failures): Network error: Failed to ping master at 127.0.107.125:38653: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)
W20250114 02:56:10.728011 3680 heartbeater.cc:643] Failed to heartbeat to 127.0.107.126:40235 (154 consecutive failures): Network error: Failed to ping master at 127.0.107.126:40235: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111)
W20250114 02:56:10.771136 3540 heartbeater.cc:643] Failed to heartbeat to 127.0.107.126:40235 (130 consecutive failures): Network error: Failed to ping master at 127.0.107.126:40235: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111)
I20250114 02:56:11.855959 5068 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:56:11.856462 5068 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:56:11.858202 5068 leader_election.cc:290] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235), bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653)
W20250114 02:56:11.861208 4075 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250114 02:56:11.863907 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235): Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111)
W20250114 02:56:11.864491 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653): Network error: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)
I20250114 02:56:11.864794 4075 leader_election.cc:304] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9a6e2eab6e541eca12ee79c1d3004e8; no voters: a8f3d9567c014741a7e628bab3e18e27, bfb2e2fecfb94af8bf892c6f774c2d89
I20250114 02:56:11.865367 5068 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250114 02:56:12.127169 3567 consensus_peers.cc:487] T c174bcbc5d51402b8ae332b87ffa2139 P 5017c179ddc640fc9ebf5dea7f3957f8 -> Peer 39c64942315c4716baaee0c251c05acd (127.0.107.66:33487): Couldn't send request to peer 39c64942315c4716baaee0c251c05acd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c174bcbc5d51402b8ae332b87ffa2139. This is attempt 31: this message will repeat every 5th retry.
I20250114 02:56:14.410073 4026 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 02:56:14.410470 4026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250114 02:56:19.413679 4026 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 02:56:19.414077 4026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250114 02:56:21.444331 5239 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:56:21.444794 5239 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:56:21.446256 5239 leader_election.cc:290] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235), bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653)
W20250114 02:56:21.448405 4075 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250114 02:56:21.451136 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235): Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111)
W20250114 02:56:21.451759 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653): Network error: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)
I20250114 02:56:21.452039 4075 leader_election.cc:304] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9a6e2eab6e541eca12ee79c1d3004e8; no voters: a8f3d9567c014741a7e628bab3e18e27, bfb2e2fecfb94af8bf892c6f774c2d89
I20250114 02:56:21.452589 5239 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250114 02:56:22.596112 3567 consensus_peers.cc:487] T c174bcbc5d51402b8ae332b87ffa2139 P 5017c179ddc640fc9ebf5dea7f3957f8 -> Peer 39c64942315c4716baaee0c251c05acd (127.0.107.66:33487): Couldn't send request to peer 39c64942315c4716baaee0c251c05acd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c174bcbc5d51402b8ae332b87ffa2139. This is attempt 36: this message will repeat every 5th retry.
I20250114 02:56:24.416574 4026 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 02:56:24.416916 4026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250114 02:56:28.842034 3532 debug-util.cc:398] Leaking SignalData structure 0x7b08000d6ac0 after lost signal to thread 3403
W20250114 02:56:28.842973 3532 debug-util.cc:398] Leaking SignalData structure 0x7b08000e2080 after lost signal to thread 3535
I20250114 02:56:29.419843 4026 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 02:56:29.420325 4026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250114 02:56:31.918980 3567 consensus_peers.cc:487] T c174bcbc5d51402b8ae332b87ffa2139 P 5017c179ddc640fc9ebf5dea7f3957f8 -> Peer 39c64942315c4716baaee0c251c05acd (127.0.107.66:33487): Couldn't send request to peer 39c64942315c4716baaee0c251c05acd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c174bcbc5d51402b8ae332b87ffa2139. This is attempt 41: this message will repeat every 5th retry.
I20250114 02:56:34.422807 4026 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 02:56:34.423154 4026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250114 02:56:35.861382 5477 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:56:35.861788 5477 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:56:35.863294 5477 leader_election.cc:290] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235), bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653)
W20250114 02:56:35.866298 4075 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250114 02:56:35.869434 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235): Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111)
W20250114 02:56:35.869933 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653): Network error: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)
I20250114 02:56:35.870154 4075 leader_election.cc:304] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9a6e2eab6e541eca12ee79c1d3004e8; no voters: a8f3d9567c014741a7e628bab3e18e27, bfb2e2fecfb94af8bf892c6f774c2d89
I20250114 02:56:35.870651 5477 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:56:39.426671 4026 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 02:56:39.427189 4026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250114 02:56:41.708905 3567 consensus_peers.cc:487] T c174bcbc5d51402b8ae332b87ffa2139 P 5017c179ddc640fc9ebf5dea7f3957f8 -> Peer 39c64942315c4716baaee0c251c05acd (127.0.107.66:33487): Couldn't send request to peer 39c64942315c4716baaee0c251c05acd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c174bcbc5d51402b8ae332b87ffa2139. This is attempt 46: this message will repeat every 5th retry.
I20250114 02:56:44.429941 4026 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 02:56:44.430241 4026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250114 02:56:44.532296 5623 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:56:44.532732 5623 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:56:44.534271 5623 leader_election.cc:290] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235), bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653)
W20250114 02:56:44.535956 4075 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250114 02:56:44.538259 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235): Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111)
W20250114 02:56:44.538816 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653): Network error: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)
I20250114 02:56:44.539040 4075 leader_election.cc:304] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9a6e2eab6e541eca12ee79c1d3004e8; no voters: a8f3d9567c014741a7e628bab3e18e27, bfb2e2fecfb94af8bf892c6f774c2d89
I20250114 02:56:44.539460 5623 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250114 02:56:45.229146 3388 debug-util.cc:398] Leaking SignalData structure 0x7b08000e52c0 after lost signal to thread 3251
W20250114 02:56:45.230000 3388 debug-util.cc:398] Leaking SignalData structure 0x7b0800045f80 after lost signal to thread 3391
I20250114 02:56:49.433300 4026 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 02:56:49.433706 4026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250114 02:56:51.529603 3674 debug-util.cc:398] Leaking SignalData structure 0x7b0800045da0 after lost signal to thread 3548
W20250114 02:56:51.530397 3674 debug-util.cc:398] Leaking SignalData structure 0x7b08000dc860 after lost signal to thread 3677
W20250114 02:56:51.691066 3567 consensus_peers.cc:487] T c174bcbc5d51402b8ae332b87ffa2139 P 5017c179ddc640fc9ebf5dea7f3957f8 -> Peer 39c64942315c4716baaee0c251c05acd (127.0.107.66:33487): Couldn't send request to peer 39c64942315c4716baaee0c251c05acd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c174bcbc5d51402b8ae332b87ffa2139. This is attempt 51: this message will repeat every 5th retry.
I20250114 02:56:52.553711 5748 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:56:52.554145 5748 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:56:52.556154 5748 leader_election.cc:290] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235), bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653)
W20250114 02:56:52.558723 4075 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250114 02:56:52.561616 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235): Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111)
W20250114 02:56:52.562136 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653): Network error: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)
I20250114 02:56:52.562425 4075 leader_election.cc:304] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9a6e2eab6e541eca12ee79c1d3004e8; no voters: a8f3d9567c014741a7e628bab3e18e27, bfb2e2fecfb94af8bf892c6f774c2d89
I20250114 02:56:52.562898 5748 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:56:54.436110 4026 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 02:56:54.436481 4026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250114 02:56:54.876401 3388 debug-util.cc:398] Leaking SignalData structure 0x7b08000ba2a0 after lost signal to thread 3251
W20250114 02:56:54.877249 3388 debug-util.cc:398] Leaking SignalData structure 0x7b08000e46c0 after lost signal to thread 3391
I20250114 02:56:59.439447 4026 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 02:56:59.439901 4026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250114 02:57:00.805239 3567 consensus_peers.cc:487] T c174bcbc5d51402b8ae332b87ffa2139 P 5017c179ddc640fc9ebf5dea7f3957f8 -> Peer 39c64942315c4716baaee0c251c05acd (127.0.107.66:33487): Couldn't send request to peer 39c64942315c4716baaee0c251c05acd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c174bcbc5d51402b8ae332b87ffa2139. This is attempt 56: this message will repeat every 5th retry.
I20250114 02:57:03.742385 5928 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:57:03.742772 5928 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:57:03.744282 5928 leader_election.cc:290] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235), bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653)
W20250114 02:57:03.746122 4075 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250114 02:57:03.748651 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235): Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111)
W20250114 02:57:03.749256 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653): Network error: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)
I20250114 02:57:03.749476 4075 leader_election.cc:304] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9a6e2eab6e541eca12ee79c1d3004e8; no voters: a8f3d9567c014741a7e628bab3e18e27, bfb2e2fecfb94af8bf892c6f774c2d89
I20250114 02:57:03.749897 5928 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:57:04.442243 4026 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 02:57:04.442582 4026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250114 02:57:09.445828 4026 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 02:57:09.446189 4026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250114 02:57:10.910858 3393 heartbeater.cc:643] Failed to heartbeat to 127.0.107.125:38653 (235 consecutive failures): Network error: Failed to ping master at 127.0.107.125:38653: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)
W20250114 02:57:11.060256 3680 heartbeater.cc:643] Failed to heartbeat to 127.0.107.126:40235 (300 consecutive failures): Network error: Failed to ping master at 127.0.107.126:40235: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111)
W20250114 02:57:11.199818 3537 heartbeater.cc:643] Failed to heartbeat to 127.0.107.125:38653 (233 consecutive failures): Network error: Failed to ping master at 127.0.107.125:38653: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)
W20250114 02:57:11.693014 3567 consensus_peers.cc:487] T c174bcbc5d51402b8ae332b87ffa2139 P 5017c179ddc640fc9ebf5dea7f3957f8 -> Peer 39c64942315c4716baaee0c251c05acd (127.0.107.66:33487): Couldn't send request to peer 39c64942315c4716baaee0c251c05acd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c174bcbc5d51402b8ae332b87ffa2139. This is attempt 61: this message will repeat every 5th retry.
W20250114 02:57:13.771461 4121 debug-util.cc:398] Leaking SignalData structure 0x7b0800054620 after lost signal to thread 4036
W20250114 02:57:13.772063 4121 debug-util.cc:398] Leaking SignalData structure 0x7b080008cce0 after lost signal to thread 4124
I20250114 02:57:14.489470 4026 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 02:57:14.489811 4026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250114 02:57:19.384616 4078 maintenance_manager.cc:418] P f9a6e2eab6e541eca12ee79c1d3004e8: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033354
I20250114 02:57:19.457134 4077 maintenance_manager.cc:642] P f9a6e2eab6e541eca12ee79c1d3004e8: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.072s user 0.048s sys 0.002s Metrics: {"bytes_written":9988,"cfile_init":1,"drs_written":1,"lbm_read_time_us":194,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2113,"lbm_writes_lt_1ms":30,"peak_mem_usage":0,"rows_written":16,"thread_start_us":424,"threads_started":1,"wal-append.queue_time_us":686}
I20250114 02:57:19.459079 4078 maintenance_manager.cc:418] P f9a6e2eab6e541eca12ee79c1d3004e8: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 976 bytes on disk
I20250114 02:57:19.460523 4077 maintenance_manager.cc:642] P f9a6e2eab6e541eca12ee79c1d3004e8: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":214,"lbm_reads_lt_1ms":4}
I20250114 02:57:19.492851 4026 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 02:57:19.493233 4026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250114 02:57:20.985216 6214 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:57:20.985635 6214 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:57:20.987284 6214 leader_election.cc:290] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235), bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653)
W20250114 02:57:20.989343 4075 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250114 02:57:20.992022 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235): Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111)
W20250114 02:57:20.992509 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653): Network error: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)
I20250114 02:57:20.992791 4075 leader_election.cc:304] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9a6e2eab6e541eca12ee79c1d3004e8; no voters: a8f3d9567c014741a7e628bab3e18e27, bfb2e2fecfb94af8bf892c6f774c2d89
I20250114 02:57:20.993278 6214 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250114 02:57:22.474217 3567 consensus_peers.cc:487] T c174bcbc5d51402b8ae332b87ffa2139 P 5017c179ddc640fc9ebf5dea7f3957f8 -> Peer 39c64942315c4716baaee0c251c05acd (127.0.107.66:33487): Couldn't send request to peer 39c64942315c4716baaee0c251c05acd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c174bcbc5d51402b8ae332b87ffa2139. This is attempt 66: this message will repeat every 5th retry.
I20250114 02:57:24.496192 4026 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 02:57:24.496608 4026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250114 02:57:29.499586 4026 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 02:57:29.499958 4026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250114 02:57:33.095333 3567 consensus_peers.cc:487] T c174bcbc5d51402b8ae332b87ffa2139 P 5017c179ddc640fc9ebf5dea7f3957f8 -> Peer 39c64942315c4716baaee0c251c05acd (127.0.107.66:33487): Couldn't send request to peer 39c64942315c4716baaee0c251c05acd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c174bcbc5d51402b8ae332b87ffa2139. This is attempt 71: this message will repeat every 5th retry.
I20250114 02:57:34.502400 4026 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 02:57:34.502774 4026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250114 02:57:35.457227 6508 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:57:35.457691 6508 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:57:35.459354 6508 leader_election.cc:290] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235), bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653)
W20250114 02:57:35.461481 4075 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250114 02:57:35.464354 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235): Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111)
W20250114 02:57:35.464926 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653): Network error: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)
I20250114 02:57:35.465162 4075 leader_election.cc:304] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9a6e2eab6e541eca12ee79c1d3004e8; no voters: a8f3d9567c014741a7e628bab3e18e27, bfb2e2fecfb94af8bf892c6f774c2d89
I20250114 02:57:35.465636 6508 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:57:39.506093 4026 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 02:57:39.506479 4026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250114 02:57:41.719944 6641 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:57:41.720429 6641 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:57:41.722010 6641 leader_election.cc:290] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235), bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653)
W20250114 02:57:41.724545 4075 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250114 02:57:41.727422 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235): Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111)
W20250114 02:57:41.727886 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653): Network error: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)
I20250114 02:57:41.728142 4075 leader_election.cc:304] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9a6e2eab6e541eca12ee79c1d3004e8; no voters: a8f3d9567c014741a7e628bab3e18e27, bfb2e2fecfb94af8bf892c6f774c2d89
I20250114 02:57:41.728770 6641 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250114 02:57:42.212558 3567 consensus_peers.cc:487] T c174bcbc5d51402b8ae332b87ffa2139 P 5017c179ddc640fc9ebf5dea7f3957f8 -> Peer 39c64942315c4716baaee0c251c05acd (127.0.107.66:33487): Couldn't send request to peer 39c64942315c4716baaee0c251c05acd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c174bcbc5d51402b8ae332b87ffa2139. This is attempt 76: this message will repeat every 5th retry.
I20250114 02:57:44.508944 4026 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 02:57:44.509277 4026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250114 02:57:46.193544 6738 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:57:46.193886 6738 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:57:46.195458 6738 leader_election.cc:290] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235), bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653)
W20250114 02:57:46.199420 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235): Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111)
W20250114 02:57:46.200057 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653): Network error: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)
I20250114 02:57:46.200335 4075 leader_election.cc:304] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9a6e2eab6e541eca12ee79c1d3004e8; no voters: a8f3d9567c014741a7e628bab3e18e27, bfb2e2fecfb94af8bf892c6f774c2d89
I20250114 02:57:46.200770 6738 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250114 02:57:48.213580 3532 debug-util.cc:398] Leaking SignalData structure 0x7b0800044500 after lost signal to thread 3403
W20250114 02:57:48.214510 3532 debug-util.cc:398] Leaking SignalData structure 0x7b08000b15e0 after lost signal to thread 3535
I20250114 02:57:49.512382 4026 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 02:57:49.512769 4026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250114 02:57:51.787528 3567 consensus_peers.cc:487] T c174bcbc5d51402b8ae332b87ffa2139 P 5017c179ddc640fc9ebf5dea7f3957f8 -> Peer 39c64942315c4716baaee0c251c05acd (127.0.107.66:33487): Couldn't send request to peer 39c64942315c4716baaee0c251c05acd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c174bcbc5d51402b8ae332b87ffa2139. This is attempt 81: this message will repeat every 5th retry.
I20250114 02:57:53.959125 6898 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:57:53.959592 6898 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:57:53.961155 6898 leader_election.cc:290] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235), bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653)
W20250114 02:57:53.963616 4075 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250114 02:57:53.966496 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235): Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111)
W20250114 02:57:53.966955 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653): Network error: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)
I20250114 02:57:53.967206 4075 leader_election.cc:304] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9a6e2eab6e541eca12ee79c1d3004e8; no voters: a8f3d9567c014741a7e628bab3e18e27, bfb2e2fecfb94af8bf892c6f774c2d89
I20250114 02:57:53.967743 6898 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:57:54.515399 4026 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 02:57:54.515736 4026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250114 02:57:59.245345 3388 debug-util.cc:398] Leaking SignalData structure 0x7b08000db720 after lost signal to thread 3251
W20250114 02:57:59.246387 3388 debug-util.cc:398] Leaking SignalData structure 0x7b08000d5be0 after lost signal to thread 3391
I20250114 02:57:59.518868 4026 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 02:57:59.519304 4026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250114 02:58:00.113303 7025 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:58:00.113824 7025 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:58:00.115542 7025 leader_election.cc:290] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235), bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653)
W20250114 02:58:00.118188 4075 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250114 02:58:00.121198 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235): Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111)
W20250114 02:58:00.121701 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653): Network error: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)
I20250114 02:58:00.124372 4075 leader_election.cc:304] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9a6e2eab6e541eca12ee79c1d3004e8; no voters: a8f3d9567c014741a7e628bab3e18e27, bfb2e2fecfb94af8bf892c6f774c2d89
I20250114 02:58:00.124974 7025 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250114 02:58:02.461694 3567 consensus_peers.cc:487] T c174bcbc5d51402b8ae332b87ffa2139 P 5017c179ddc640fc9ebf5dea7f3957f8 -> Peer 39c64942315c4716baaee0c251c05acd (127.0.107.66:33487): Couldn't send request to peer 39c64942315c4716baaee0c251c05acd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c174bcbc5d51402b8ae332b87ffa2139. This is attempt 86: this message will repeat every 5th retry.
W20250114 02:58:03.305430 3674 debug-util.cc:398] Leaking SignalData structure 0x7b08000db700 after lost signal to thread 3548
W20250114 02:58:03.306411 3674 debug-util.cc:398] Leaking SignalData structure 0x7b08000cea00 after lost signal to thread 3677
I20250114 02:58:04.521977 4026 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 02:58:04.522434 4026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250114 02:58:07.822664 7164 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:58:07.823055 7164 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:58:07.824710 7164 leader_election.cc:290] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235), bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653)
W20250114 02:58:07.826520 4075 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250114 02:58:07.828840 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235): Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111)
W20250114 02:58:07.829499 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653): Network error: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)
I20250114 02:58:07.829756 4075 leader_election.cc:304] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9a6e2eab6e541eca12ee79c1d3004e8; no voters: a8f3d9567c014741a7e628bab3e18e27, bfb2e2fecfb94af8bf892c6f774c2d89
I20250114 02:58:07.830185 7164 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:58:09.525828 4026 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 02:58:09.526207 4026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250114 02:58:11.309391 3398 heartbeater.cc:643] Failed to heartbeat to 127.0.107.126:40235 (369 consecutive failures): Network error: Failed to ping master at 127.0.107.126:40235: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111)
W20250114 02:58:11.446363 3680 heartbeater.cc:643] Failed to heartbeat to 127.0.107.126:40235 (449 consecutive failures): Network error: Failed to ping master at 127.0.107.126:40235: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111)
W20250114 02:58:11.603322 3540 heartbeater.cc:643] Failed to heartbeat to 127.0.107.126:40235 (370 consecutive failures): Network error: Failed to ping master at 127.0.107.126:40235: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111)
W20250114 02:58:13.205161 3567 consensus_peers.cc:487] T c174bcbc5d51402b8ae332b87ffa2139 P 5017c179ddc640fc9ebf5dea7f3957f8 -> Peer 39c64942315c4716baaee0c251c05acd (127.0.107.66:33487): Couldn't send request to peer 39c64942315c4716baaee0c251c05acd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c174bcbc5d51402b8ae332b87ffa2139. This is attempt 91: this message will repeat every 5th retry.
I20250114 02:58:14.528436 4026 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 02:58:14.528753 4026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250114 02:58:19.531912 4026 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 02:58:19.532368 4026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250114 02:58:22.920689 7467 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:58:22.921128 7467 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:58:22.922921 7467 leader_election.cc:290] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235), bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653)
W20250114 02:58:22.925271 4075 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250114 02:58:22.929677 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235): Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111)
W20250114 02:58:22.930413 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653): Network error: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)
I20250114 02:58:22.930677 4075 leader_election.cc:304] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9a6e2eab6e541eca12ee79c1d3004e8; no voters: a8f3d9567c014741a7e628bab3e18e27, bfb2e2fecfb94af8bf892c6f774c2d89
I20250114 02:58:22.931181 7467 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250114 02:58:23.211407 3567 consensus_peers.cc:487] T c174bcbc5d51402b8ae332b87ffa2139 P 5017c179ddc640fc9ebf5dea7f3957f8 -> Peer 39c64942315c4716baaee0c251c05acd (127.0.107.66:33487): Couldn't send request to peer 39c64942315c4716baaee0c251c05acd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c174bcbc5d51402b8ae332b87ffa2139. This is attempt 96: this message will repeat every 5th retry.
I20250114 02:58:24.535205 4026 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 02:58:24.535507 4026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250114 02:58:29.538687 4026 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 02:58:29.539104 4026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250114 02:58:31.600648 4121 debug-util.cc:398] Leaking SignalData structure 0x7b08000b3ec0 after lost signal to thread 4036
W20250114 02:58:31.601159 4121 debug-util.cc:398] Leaking SignalData structure 0x7b080008dbe0 after lost signal to thread 4124
I20250114 02:58:32.374169 7650 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:58:32.374614 7650 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:58:32.376039 7650 leader_election.cc:290] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235), bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653)
W20250114 02:58:32.378341 4075 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250114 02:58:32.381032 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235): Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111)
W20250114 02:58:32.381464 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653): Network error: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)
I20250114 02:58:32.381700 4075 leader_election.cc:304] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9a6e2eab6e541eca12ee79c1d3004e8; no voters: a8f3d9567c014741a7e628bab3e18e27, bfb2e2fecfb94af8bf892c6f774c2d89
I20250114 02:58:32.382174 7650 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250114 02:58:33.770967 3567 consensus_peers.cc:487] T c174bcbc5d51402b8ae332b87ffa2139 P 5017c179ddc640fc9ebf5dea7f3957f8 -> Peer 39c64942315c4716baaee0c251c05acd (127.0.107.66:33487): Couldn't send request to peer 39c64942315c4716baaee0c251c05acd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c174bcbc5d51402b8ae332b87ffa2139. This is attempt 101: this message will repeat every 5th retry.
I20250114 02:58:34.577761 4026 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 02:58:34.578172 4026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250114 02:58:39.581167 4026 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 02:58:39.581568 4026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250114 02:58:43.993582 3567 consensus_peers.cc:487] T c174bcbc5d51402b8ae332b87ffa2139 P 5017c179ddc640fc9ebf5dea7f3957f8 -> Peer 39c64942315c4716baaee0c251c05acd (127.0.107.66:33487): Couldn't send request to peer 39c64942315c4716baaee0c251c05acd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c174bcbc5d51402b8ae332b87ffa2139. This is attempt 106: this message will repeat every 5th retry.
I20250114 02:58:44.584314 4026 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 02:58:44.584676 4026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250114 02:58:49.587841 4026 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 02:58:49.588279 4026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250114 02:58:52.197916 8037 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:58:52.198469 8037 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:58:52.200335 8037 leader_election.cc:290] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235), bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653)
W20250114 02:58:52.202530 4075 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250114 02:58:52.205423 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235): Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111)
W20250114 02:58:52.205914 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653): Network error: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)
I20250114 02:58:52.206140 4075 leader_election.cc:304] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9a6e2eab6e541eca12ee79c1d3004e8; no voters: a8f3d9567c014741a7e628bab3e18e27, bfb2e2fecfb94af8bf892c6f774c2d89
I20250114 02:58:52.206627 8037 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250114 02:58:54.547317 3567 consensus_peers.cc:487] T c174bcbc5d51402b8ae332b87ffa2139 P 5017c179ddc640fc9ebf5dea7f3957f8 -> Peer 39c64942315c4716baaee0c251c05acd (127.0.107.66:33487): Couldn't send request to peer 39c64942315c4716baaee0c251c05acd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c174bcbc5d51402b8ae332b87ffa2139. This is attempt 111: this message will repeat every 5th retry.
I20250114 02:58:54.590734 4026 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 02:58:54.591063 4026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250114 02:58:55.107504 8101 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:58:55.107864 8101 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:58:55.109280 8101 leader_election.cc:290] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235), bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653)
W20250114 02:58:55.113425 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235): Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111)
W20250114 02:58:55.114094 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653): Network error: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)
I20250114 02:58:55.114429 4075 leader_election.cc:304] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9a6e2eab6e541eca12ee79c1d3004e8; no voters: a8f3d9567c014741a7e628bab3e18e27, bfb2e2fecfb94af8bf892c6f774c2d89
I20250114 02:58:55.114874 8101 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250114 02:58:57.583829 3532 debug-util.cc:398] Leaking SignalData structure 0x7b08000d5f80 after lost signal to thread 3403
W20250114 02:58:57.585187 3532 debug-util.cc:398] Leaking SignalData structure 0x7b08000c3fa0 after lost signal to thread 3535
I20250114 02:58:59.594344 4026 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 02:58:59.594795 4026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250114 02:59:04.597373 4026 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 02:59:04.597712 4026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250114 02:59:05.414258 3567 consensus_peers.cc:487] T c174bcbc5d51402b8ae332b87ffa2139 P 5017c179ddc640fc9ebf5dea7f3957f8 -> Peer 39c64942315c4716baaee0c251c05acd (127.0.107.66:33487): Couldn't send request to peer 39c64942315c4716baaee0c251c05acd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c174bcbc5d51402b8ae332b87ffa2139. This is attempt 116: this message will repeat every 5th retry.
I20250114 02:59:06.045607 8321 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:59:06.046192 8321 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:59:06.047658 8321 leader_election.cc:290] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235), bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653)
W20250114 02:59:06.050125 4075 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250114 02:59:06.053223 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235): Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111)
W20250114 02:59:06.053663 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653): Network error: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)
I20250114 02:59:06.053905 4075 leader_election.cc:304] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9a6e2eab6e541eca12ee79c1d3004e8; no voters: a8f3d9567c014741a7e628bab3e18e27, bfb2e2fecfb94af8bf892c6f774c2d89
I20250114 02:59:06.054425 8321 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:59:09.600972 4026 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 02:59:09.601377 4026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250114 02:59:11.462073 3680 heartbeater.cc:643] Failed to heartbeat to 127.0.107.126:40235 (599 consecutive failures): Network error: Failed to ping master at 127.0.107.126:40235: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111)
W20250114 02:59:11.688498 3398 heartbeater.cc:643] Failed to heartbeat to 127.0.107.126:40235 (489 consecutive failures): Network error: Failed to ping master at 127.0.107.126:40235: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111)
W20250114 02:59:12.027767 3537 heartbeater.cc:643] Failed to heartbeat to 127.0.107.125:38653 (473 consecutive failures): Network error: Failed to ping master at 127.0.107.125:38653: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)
I20250114 02:59:13.097117 8465 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:59:13.097653 8465 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:59:13.099166 8465 leader_election.cc:290] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235), bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653)
W20250114 02:59:13.101744 4075 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250114 02:59:13.104702 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235): Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111)
W20250114 02:59:13.105331 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653): Network error: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)
I20250114 02:59:13.105589 4075 leader_election.cc:304] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9a6e2eab6e541eca12ee79c1d3004e8; no voters: a8f3d9567c014741a7e628bab3e18e27, bfb2e2fecfb94af8bf892c6f774c2d89
I20250114 02:59:13.106055 8465 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:59:14.604359 4026 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 02:59:14.604754 4026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250114 02:59:15.062342 3567 consensus_peers.cc:487] T c174bcbc5d51402b8ae332b87ffa2139 P 5017c179ddc640fc9ebf5dea7f3957f8 -> Peer 39c64942315c4716baaee0c251c05acd (127.0.107.66:33487): Couldn't send request to peer 39c64942315c4716baaee0c251c05acd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c174bcbc5d51402b8ae332b87ffa2139. This is attempt 121: this message will repeat every 5th retry.
I20250114 02:59:19.607952 4026 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 02:59:19.608428 4026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250114 02:59:20.398653 8609 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:59:20.399119 8609 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:59:20.400561 8609 leader_election.cc:290] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235), bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653)
W20250114 02:59:20.402789 4075 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250114 02:59:20.405542 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235): Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111)
W20250114 02:59:20.406209 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653): Network error: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)
I20250114 02:59:20.406427 4075 leader_election.cc:304] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9a6e2eab6e541eca12ee79c1d3004e8; no voters: a8f3d9567c014741a7e628bab3e18e27, bfb2e2fecfb94af8bf892c6f774c2d89
I20250114 02:59:20.406895 8609 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250114 02:59:22.535802 3674 debug-util.cc:398] Leaking SignalData structure 0x7b08000d95c0 after lost signal to thread 3548
W20250114 02:59:22.536623 3674 debug-util.cc:398] Leaking SignalData structure 0x7b08000dabc0 after lost signal to thread 3677
I20250114 02:59:24.610832 4026 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 02:59:24.611173 4026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250114 02:59:25.090682 3567 consensus_peers.cc:487] T c174bcbc5d51402b8ae332b87ffa2139 P 5017c179ddc640fc9ebf5dea7f3957f8 -> Peer 39c64942315c4716baaee0c251c05acd (127.0.107.66:33487): Couldn't send request to peer 39c64942315c4716baaee0c251c05acd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c174bcbc5d51402b8ae332b87ffa2139. This is attempt 126: this message will repeat every 5th retry.
I20250114 02:59:29.614878 4026 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 02:59:29.615399 4026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250114 02:59:34.540055 3567 consensus_peers.cc:487] T c174bcbc5d51402b8ae332b87ffa2139 P 5017c179ddc640fc9ebf5dea7f3957f8 -> Peer 39c64942315c4716baaee0c251c05acd (127.0.107.66:33487): Couldn't send request to peer 39c64942315c4716baaee0c251c05acd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c174bcbc5d51402b8ae332b87ffa2139. This is attempt 131: this message will repeat every 5th retry.
I20250114 02:59:34.618207 4026 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 02:59:34.618534 4026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250114 02:59:37.054383 3532 debug-util.cc:398] Leaking SignalData structure 0x7b08000c0f00 after lost signal to thread 3403
W20250114 02:59:37.055721 3532 debug-util.cc:398] Leaking SignalData structure 0x7b08000b1fc0 after lost signal to thread 3535
I20250114 02:59:39.014493 8977 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:59:39.015055 8977 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:59:39.017577 8977 leader_election.cc:290] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235), bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653)
W20250114 02:59:39.020401 4075 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250114 02:59:39.023159 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235): Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111)
W20250114 02:59:39.023717 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653): Network error: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)
I20250114 02:59:39.023998 4075 leader_election.cc:304] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9a6e2eab6e541eca12ee79c1d3004e8; no voters: a8f3d9567c014741a7e628bab3e18e27, bfb2e2fecfb94af8bf892c6f774c2d89
I20250114 02:59:39.024559 8977 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:59:39.577775 8992 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:59:39.578104 8992 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:59:39.579528 8992 leader_election.cc:290] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235), bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653)
W20250114 02:59:39.583662 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235): Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111)
W20250114 02:59:39.584547 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653): Network error: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)
I20250114 02:59:39.584875 4075 leader_election.cc:304] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9a6e2eab6e541eca12ee79c1d3004e8; no voters: a8f3d9567c014741a7e628bab3e18e27, bfb2e2fecfb94af8bf892c6f774c2d89
I20250114 02:59:39.585803 8992 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:59:39.620990 4026 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 02:59:39.621369 4026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250114 02:59:41.384949 3388 debug-util.cc:398] Leaking SignalData structure 0x7b08000da620 after lost signal to thread 3251
W20250114 02:59:41.385689 3388 debug-util.cc:398] Leaking SignalData structure 0x7b08000445a0 after lost signal to thread 3391
I20250114 02:59:44.623993 4026 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 02:59:44.624379 4026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250114 02:59:44.693105 3532 debug-util.cc:398] Leaking SignalData structure 0x7b08000b63c0 after lost signal to thread 3403
W20250114 02:59:44.693912 3532 debug-util.cc:398] Leaking SignalData structure 0x7b08000734c0 after lost signal to thread 3535
W20250114 02:59:44.996157 3567 consensus_peers.cc:487] T c174bcbc5d51402b8ae332b87ffa2139 P 5017c179ddc640fc9ebf5dea7f3957f8 -> Peer 39c64942315c4716baaee0c251c05acd (127.0.107.66:33487): Couldn't send request to peer 39c64942315c4716baaee0c251c05acd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c174bcbc5d51402b8ae332b87ffa2139. This is attempt 136: this message will repeat every 5th retry.
W20250114 02:59:48.286664 3674 debug-util.cc:398] Leaking SignalData structure 0x7b08000d9300 after lost signal to thread 3548
W20250114 02:59:48.287376 3674 debug-util.cc:398] Leaking SignalData structure 0x7b08000d9740 after lost signal to thread 3677
I20250114 02:59:49.628548 4026 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 02:59:49.629081 4026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250114 02:59:52.137991 4121 debug-util.cc:398] Leaking SignalData structure 0x7b080009c5a0 after lost signal to thread 4036
W20250114 02:59:52.138528 4121 debug-util.cc:398] Leaking SignalData structure 0x7b080003a9e0 after lost signal to thread 4124
W20250114 02:59:54.491339 3567 consensus_peers.cc:487] T c174bcbc5d51402b8ae332b87ffa2139 P 5017c179ddc640fc9ebf5dea7f3957f8 -> Peer 39c64942315c4716baaee0c251c05acd (127.0.107.66:33487): Couldn't send request to peer 39c64942315c4716baaee0c251c05acd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c174bcbc5d51402b8ae332b87ffa2139. This is attempt 141: this message will repeat every 5th retry.
I20250114 02:59:54.546533 9289 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 02:59:54.546985 9289 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 02:59:54.548723 9289 leader_election.cc:290] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235), bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653)
W20250114 02:59:54.550709 4075 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250114 02:59:54.552953 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235): Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111)
W20250114 02:59:54.553566 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653): Network error: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)
I20250114 02:59:54.553797 4075 leader_election.cc:304] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9a6e2eab6e541eca12ee79c1d3004e8; no voters: a8f3d9567c014741a7e628bab3e18e27, bfb2e2fecfb94af8bf892c6f774c2d89
I20250114 02:59:54.554275 9289 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 02:59:54.661953 4026 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 02:59:54.662329 4026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250114 02:59:59.665638 4026 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 02:59:59.666044 4026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250114 03:00:03.877288 3567 consensus_peers.cc:487] T c174bcbc5d51402b8ae332b87ffa2139 P 5017c179ddc640fc9ebf5dea7f3957f8 -> Peer 39c64942315c4716baaee0c251c05acd (127.0.107.66:33487): Couldn't send request to peer 39c64942315c4716baaee0c251c05acd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c174bcbc5d51402b8ae332b87ffa2139. This is attempt 146: this message will repeat every 5th retry.
I20250114 03:00:04.668546 4026 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 03:00:04.668844 4026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250114 03:00:05.503494 9507 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 03:00:05.503991 9507 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 03:00:05.505688 9507 leader_election.cc:290] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235), bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653)
W20250114 03:00:05.507795 4075 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250114 03:00:05.510353 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235): Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111)
W20250114 03:00:05.510939 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653): Network error: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)
I20250114 03:00:05.511199 4075 leader_election.cc:304] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9a6e2eab6e541eca12ee79c1d3004e8; no voters: a8f3d9567c014741a7e628bab3e18e27, bfb2e2fecfb94af8bf892c6f774c2d89
I20250114 03:00:05.511672 9507 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 03:00:09.672284 4026 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 03:00:09.672761 4026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250114 03:00:11.797281 3679 heartbeater.cc:643] Failed to heartbeat to 127.0.107.125:38653 (734 consecutive failures): Network error: Failed to ping master at 127.0.107.125:38653: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)
W20250114 03:00:12.095798 3398 heartbeater.cc:643] Failed to heartbeat to 127.0.107.126:40235 (609 consecutive failures): Network error: Failed to ping master at 127.0.107.126:40235: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111)
W20250114 03:00:12.443877 3537 heartbeater.cc:643] Failed to heartbeat to 127.0.107.125:38653 (593 consecutive failures): Network error: Failed to ping master at 127.0.107.125:38653: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)
W20250114 03:00:12.878360 3567 consensus_peers.cc:487] T c174bcbc5d51402b8ae332b87ffa2139 P 5017c179ddc640fc9ebf5dea7f3957f8 -> Peer 39c64942315c4716baaee0c251c05acd (127.0.107.66:33487): Couldn't send request to peer 39c64942315c4716baaee0c251c05acd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c174bcbc5d51402b8ae332b87ffa2139. This is attempt 151: this message will repeat every 5th retry.
I20250114 03:00:14.675719 4026 master_runner.cc:186] Error getting master registration for 127.0.107.126:40235: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111),
I20250114 03:00:14.676136 4026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250114 03:00:14.930585 9692 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 03:00:14.931119 9692 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f3d9567c014741a7e628bab3e18e27" member_type: VOTER last_known_addr { host: "127.0.107.126" port: 40235 } } peers { permanent_uuid: "bfb2e2fecfb94af8bf892c6f774c2d89" member_type: VOTER last_known_addr { host: "127.0.107.125" port: 38653 } } peers { permanent_uuid: "f9a6e2eab6e541eca12ee79c1d3004e8" member_type: VOTER last_known_addr { host: "127.0.107.124" port: 44431 } }
I20250114 03:00:14.932775 9692 leader_election.cc:290] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235), bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653)
W20250114 03:00:14.934937 4075 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250114 03:00:14.937958 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a8f3d9567c014741a7e628bab3e18e27 (127.0.107.126:40235): Network error: Client connection negotiation failed: client connection to 127.0.107.126:40235: connect: Connection refused (error 111)
W20250114 03:00:14.938547 4075 leader_election.cc:336] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bfb2e2fecfb94af8bf892c6f774c2d89 (127.0.107.125:38653): Network error: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)
I20250114 03:00:14.938783 4075 leader_election.cc:304] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9a6e2eab6e541eca12ee79c1d3004e8; no voters: a8f3d9567c014741a7e628bab3e18e27, bfb2e2fecfb94af8bf892c6f774c2d89
I20250114 03:00:14.939334 9692 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9a6e2eab6e541eca12ee79c1d3004e8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 03:00:15.095688 4078 maintenance_manager.cc:418] P f9a6e2eab6e541eca12ee79c1d3004e8: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 976 bytes on disk
I20250114 03:00:15.107959 4077 maintenance_manager.cc:642] P f9a6e2eab6e541eca12ee79c1d3004e8: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.011s user 0.011s sys 0.000s Metrics: {"dirs.queue_time_us":1366,"dirs.run_cpu_time_us":213,"dirs.run_wall_time_us":214,"thread_start_us":1663,"threads_started":2,"wal-append.queue_time_us":939}
W20250114 03:00:15.449895 3690 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.0.107.126:40235,127.0.107.125:38653,127.0.107.124:44431, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)
W20250114 03:00:15.450382 3690 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.0.107.126:40235,127.0.107.125:38653,127.0.107.124:44431, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)
W20250114 03:00:15.450670 3690 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.0.107.126:40235,127.0.107.125:38653,127.0.107.124:44431, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)
W20250114 03:00:15.451102 3690 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.0.107.126:40235,127.0.107.125:38653,127.0.107.124:44431, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)
F20250114 03:00:15.462551 3696 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c27b5989e4ba41f285a14120cb0dba75 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.107.126:40235,127.0.107.125:38653,127.0.107.124:44431, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)
F20250114 03:00:15.464116 3701 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.107.126:40235,127.0.107.125:38653,127.0.107.124:44431, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)
*** Check failure stack trace: ***
*** Aborted at 1736823615 (unix time) try "date -d @1736823615" if you are using GNU date ***
*** Check failure stack trace: ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e8000001ad) received by PID 429 (TID 0x7fcfa5cfe700) from PID 429; stack trace: ***
@ 0x2dac70 __tsan::CallUserSignalHandler() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1915
@ 0x2dd153 rtl_sigaction() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:2005
@ 0x7fcfb7577980 (unknown) at ??:0
F20250114 03:00:15.464116 3701 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.107.126:40235,127.0.107.125:38653,127.0.107.124:44431, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)F20250114 03:00:15.482082 3700 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.107.126:40235,127.0.107.125:38653,127.0.107.124:44431, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)
*** Check failure stack trace: ***
@ 0x7fcfb2791fb7 gsignal at ??:0
@ 0x7fcfb2793921 abort at ??:0
@ 0x2e0c66 abort at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:263
@ 0x7fcfb584cda4 google::LogMessage::SendToLog() at ??:0
@ 0x7fcfb584d950 google::LogMessage::Flush() at ??:0
@ 0x7fcfb5852a8b google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x35cda4 kudu::MasterStressTest::CreateTableThread() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/master-stress-test.cc:?
@ 0x38abbe _ZNSt3__18__invokeIMN4kudu16MasterStressTestEFvvEPNS1_26MasterStressTest_Test_TestEJEvEEDTcldsdeclsr3std3__1E7forwardIT0_Efp0_Efp_spclsr3std3__1E7forwardIT1_Efp1_EEEOT_OS7_DpOS8_ at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/type_traits:3840
@ 0x38aa71 _ZNSt3__116__thread_executeINS_10unique_ptrINS_15__thread_structENS_14default_deleteIS2_EEEEMN4kudu16MasterStressTestEFvvEJPNS6_26MasterStressTest_Test_TestEEJLm2EEEEvRNS_5tupleIJT_T0_DpT1_EEENS_15__tuple_indicesIJXspT2_EEEE at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/thread:281
@ 0x38a1bb std::__1::__thread_proxy<>() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/thread:?
@ 0x2dab30 __tsan_thread_start_func at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:959
@ 0x7fcfb756c6db start_thread at ??:0
F20250114 03:00:15.464116 3701 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.107.126:40235,127.0.107.125:38653,127.0.107.124:44431, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)F20250114 03:00:15.482082 3700 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.107.126:40235,127.0.107.125:38653,127.0.107.124:44431, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)F000F20250114 03:00:15.473937 3704 master-stress-test.cc:391] 000F20250114 03:00:15.471597 3705 master-stress-test.cc:391] 000F20250114 03:00:15.472332 3697 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1e2047182d2241f4b619010ca10e11fb on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.107.126:40235,127.0.107.125:38653,127.0.107.124:44431, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250114 03:00:15.464116 3701 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.107.126:40235,127.0.107.125:38653,127.0.107.124:44431, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)F20250114 03:00:15.482082 3700 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.107.126:40235,127.0.107.125:38653,127.0.107.124:44431, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)F000F20250114 03:00:15.473937 3704 master-stress-test.cc:391] 000F20250114 03:00:15.471597 3705 master-stress-test.cc:391] 000F20250114 03:00:15.472332 3697 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_1e2047182d2241f4b619010ca10e11fb on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.107.126:40235,127.0.107.125:38653,127.0.107.124:44431, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)F2025 114 3: 0:15.483904 3699 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f39207a74e3a4f68acdb67e4a70f7510 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.107.126:40235,127.0.107.125:38653,127.0.107.124:44431, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.107.125:38653: connect: Connection refused (error 111)
*** Check failure stack trace: ***
@ 0x7fcfb287471f clone at ??:0